2024-09-14T06:17:41.804 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-14T06:17:41.826 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T06:17:41.915 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380 branch: main description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900380' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5839 sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 sleep_before_teardown: 0 subset: 99/128 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/1YN9cdEByEgxDm3lqN2v8GTzP/mHK4T/7oMWEBifSOh9bmZZXO3sHGPVW1dKEYnMLNg2KODvzIA8Wq6ZWRz0= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+vaI4OBZX+ycCfdIr6NI27KSkRiUV6FBKOMObmKr/VHKf+FKu+UhLydleKQ0YcZ1uFNsOnPH7i/3IMTt+oxTw= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' RBD_MIRROR_MIN_COMPAT_CLIENT: octopus RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_20:16:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-14T06:17:41.915 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa; will attempt to use it 2024-09-14T06:17:41.916 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa/tasks 2024-09-14T06:17:41.916 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-14T06:17:41.918 INFO:teuthology.task.internal:Checking packages... 2024-09-14T06:17:41.940 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7f24e589328ae9e639c0009d93340e5b235d0a47' 2024-09-14T06:17:41.940 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T06:17:41.940 INFO:teuthology.packaging:ref: None 2024-09-14T06:17:41.940 INFO:teuthology.packaging:tag: None 2024-09-14T06:17:41.940 INFO:teuthology.packaging:branch: main 2024-09-14T06:17:41.940 INFO:teuthology.packaging:sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T06:17:41.941 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-14T06:17:42.156 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4926.g1c941a66 2024-09-14T06:17:42.157 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-14T06:17:42.164 INFO:teuthology.task.internal:no buildpackages task found 2024-09-14T06:17:42.164 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-14T06:17:42.180 INFO:teuthology.task.internal:Saving configuration 2024-09-14T06:17:42.196 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-14T06:17:42.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-14T06:17:42.228 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 06:12:46.638645', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/1YN9cdEByEgxDm3lqN2v8GTzP/mHK4T/7oMWEBifSOh9bmZZXO3sHGPVW1dKEYnMLNg2KODvzIA8Wq6ZWRz0='} 2024-09-14T06:17:42.248 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 06:12:46.639958', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+vaI4OBZX+ycCfdIr6NI27KSkRiUV6FBKOMObmKr/VHKf+FKu+UhLydleKQ0YcZ1uFNsOnPH7i/3IMTt+oxTw='} 2024-09-14T06:17:42.248 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-14T06:17:42.254 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-14T06:17:42.254 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-14T06:17:42.254 INFO:teuthology.run_tasks:Running task console_log... 2024-09-14T06:17:42.374 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f601c527be0>, signals=[15]) 2024-09-14T06:17:42.374 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-14T06:17:42.380 INFO:teuthology.task.internal:Opening connections... 2024-09-14T06:17:42.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T06:17:42.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:17:42.460 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:17:42.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:17:42.534 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-14T06:17:42.541 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-09-14T06:17:42.559 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-09-14T06:17:42.560 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-09-14T06:17:42.617 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-09-14T06:17:42.617 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T06:17:42.618 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T06:17:42.619 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-09-14T06:17:42.635 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-09-14T06:17:42.654 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-09-14T06:17:42.654 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-09-14T06:17:42.710 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T06:17:42.711 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T06:17:42.712 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-09-14T06:17:42.727 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-14T06:17:42.736 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-14T06:17:42.741 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-14T06:17:42.742 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-09-14T06:17:42.744 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-09-14T06:17:42.768 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-14T06:17:42.773 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-14T06:17:42.773 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-09-14T06:17:42.801 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-09-14T06:17:42.927 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-14T06:17:42.983 INFO:teuthology.run_tasks:Running task kernel... 2024-09-14T06:17:43.044 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-14T06:17:43.045 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-14T06:17:43.045 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T06:17:43.045 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T06:17:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:17:43.062 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-14T06:17:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:17:43.063 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-09-14T06:17:43.120 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-503.el9.x86_64 2024-09-14T06:17:43.120 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-503.el9.x86_64 2024-09-14T06:17:43.121 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-09-14T06:17:43.121 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-503.el9.x86_64 2024-09-14T06:17:43.122 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-503.el9.x86_64 2024-09-14T06:17:43.122 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-09-14T06:17:46.294 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.2 MB 00:01 2024-09-14T06:17:47.232 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.2 MB 00:02 2024-09-14T06:17:49.911 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 20 MB 00:02 2024-09-14T06:17:53.749 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 20 MB 00:05 2024-09-14T06:17:54.429 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.5 MB 00:00 2024-09-14T06:17:56.879 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-14T06:17:57.984 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-14T06:17:59.316 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.5 MB 00:01 2024-09-14T06:18:01.000 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 19 kB 00:00 2024-09-14T06:18:02.096 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-14T06:18:03.389 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-14T06:18:05.404 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T06:18:05.404 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T06:18:05.487 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:18:05.499 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:18:05.500 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-09-14T06:18:05.501 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:18:05.501 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 79 M 2024-09-14T06:18:05.501 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 126 M 2024-09-14T06:18:05.501 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:18:05.749 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 224 kB/s | 23 kB 00:00 2024-09-14T06:18:06.133 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-14T06:18:06.567 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 38 MB/s | 35 MB 00:00 2024-09-14T06:18:06.834 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-14T06:18:06.834 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:18:06.834 INFO:teuthology.orchestra.run.smithi170.stdout:Total 59 MB/s | 79 MB 00:01 2024-09-14T06:18:07.512 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:18:07.810 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-14T06:18:07.844 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:18:07.844 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:18:08.015 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:01 ago on Sat 14 Sep 2024 06:18:07 AM UTC. 2024-09-14T06:18:08.807 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:18:08.807 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:18:09.917 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T06:18:09.917 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T06:18:10.012 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:18:10.024 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:18:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-14T06:18:10.024 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:18:10.024 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:18:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T06:18:10.024 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:18:10.024 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T06:18:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T06:18:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T06:18:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:18:10.025 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:18:10.025 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:18:10.025 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-09-14T06:18:10.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:18:10.026 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 79 M 2024-09-14T06:18:10.026 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 126 M 2024-09-14T06:18:10.026 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:18:10.170 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 233 kB/s | 23 kB 00:00 2024-09-14T06:18:10.570 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-14T06:18:11.045 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 33 MB/s | 29 MB 00:00 2024-09-14T06:18:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:18:11.354 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-14T06:18:11.354 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:18:11.354 INFO:teuthology.orchestra.run.smithi001.stdout:Total 59 MB/s | 79 MB 00:01 2024-09-14T06:18:11.906 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T06:18:11.984 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T06:18:12.066 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:18:12.472 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:18:12.473 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:18:13.466 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:18:13.467 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:18:14.773 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T06:18:14.905 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T06:18:16.046 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:18:16.713 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T06:18:16.796 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T06:18:19.586 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T06:18:19.713 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T06:18:19.724 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T06:18:19.755 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:18:24.619 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:18:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T06:18:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:18:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:02.483 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:02.522 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T06:19:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T06:19:02.897 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:19:03.733 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:19:04.089 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T06:19:04.178 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T06:19:04.180 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T06:19:04.180 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T06:19:04.180 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T06:19:04.180 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T06:19:04.180 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-14T06:19:04.180 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T06:19:04.180 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T06:19:04.239 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T06:19:04.239 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T06:19:04.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-09-14T06:19:04.271 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-09-14T06:19:04.271 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-09-14T06:19:04.271 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-09-14T06:19:04.271 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 82.el9 2024-09-14T06:19:04.271 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8274670 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-09-14T06:19:04.272 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-09-14T06:19:04.273 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:19:04.273 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T06:19:04.273 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T06:19:04.273 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T06:19:04.273 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-09-14T06:19:04.273 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:19:04.273 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T06:19:04.274 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T06:19:04.274 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-505.el9.x86_64 2024-09-14T06:19:04.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T06:19:04.947 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-09-14T06:19:06.365 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T06:19:06.391 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-09-14T06:19:06.402 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-09-14T06:19:06.457 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.27fEmNdhjl 2024-09-14T06:19:06.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.27fEmNdhjl 2024-09-14T06:19:06.525 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.27fEmNdhjl 2024-09-14T06:19:06.658 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.27fEmNdhjl is 6KB 2024-09-14T06:19:06.707 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.27fEmNdhjl 2024-09-14T06:19:06.721 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T06:19:06.786 INFO:teuthology.orchestra.run.smithi170.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T06:19:06.786 INFO:teuthology.orchestra.run.smithi170.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T06:19:06.786 INFO:teuthology.orchestra.run.smithi170.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T06:19:06.786 INFO:teuthology.orchestra.run.smithi170.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T06:19:06.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T06:19:06.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-09-14T06:19:09.301 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:09.342 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T06:19:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T06:19:09.721 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T06:19:10.625 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:19:10.626 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:19:10.965 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T06:19:11.053 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T06:19:11.053 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T06:19:11.053 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T06:19:11.053 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T06:19:11.053 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T06:19:11.054 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-14T06:19:11.054 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T06:19:11.054 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T06:19:11.112 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T06:19:11.112 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T06:19:11.113 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 82.el9 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8274670 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-09-14T06:19:11.150 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-09-14T06:19:11.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:19:11.152 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T06:19:11.152 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T06:19:11.152 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-505.el9.x86_64 2024-09-14T06:19:11.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T06:19:11.813 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-09-14T06:19:13.049 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T06:19:13.074 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-09-14T06:19:13.075 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-14T06:19:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.ME5kwOLg4E 2024-09-14T06:19:13.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ME5kwOLg4E 2024-09-14T06:19:13.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.ME5kwOLg4E 2024-09-14T06:19:13.293 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.ME5kwOLg4E is 6KB 2024-09-14T06:19:13.342 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.ME5kwOLg4E 2024-09-14T06:19:13.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T06:19:13.419 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T06:19:13.419 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T06:19:13.419 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T06:19:13.419 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T06:19:13.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T06:19:13.559 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-09-14T06:19:36.973 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-14T06:19:36.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-09-14T06:19:36.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:19:43.566 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-14T06:19:43.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-14T06:19:43.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:19:55.329 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-09-14T06:20:01.981 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-09-14T06:20:04.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-09-14T06:20:04.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:20:07.389 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-09-14T06:20:10.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-14T06:20:10.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:20:19.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-09-14T06:20:19.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:20:51.455 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-09-14T06:20:52.153 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-09-14T06:20:52.153 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-14T06:20:52.153 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-09-14T06:20:52.168 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-505.el9.x86_64 2024-09-14T06:20:52.168 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T06:20:52.168 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T06:20:52.168 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T06:20:53.169 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-14T06:20:53.169 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T06:20:53.263 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-09-14T06:20:53.283 DEBUG:teuthology.parallel:result is None 2024-09-14T06:21:10.989 DEBUG:teuthology.orchestra.remote:timed out 2024-09-14T06:21:22.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-14T06:21:22.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:21:23.261 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-14T06:21:23.644 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-14T06:21:23.644 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-14T06:21:23.644 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-14T06:21:23.661 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-505.el9.x86_64 2024-09-14T06:21:23.662 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T06:21:23.662 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T06:21:23.662 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T06:21:24.662 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-14T06:21:24.663 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T06:21:24.755 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-09-14T06:21:24.781 DEBUG:teuthology.parallel:result is None 2024-09-14T06:21:24.782 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-14T06:21:24.788 INFO:teuthology.task.internal:Creating test directory... 2024-09-14T06:21:24.789 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T06:21:24.791 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T06:21:24.809 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-14T06:21:24.816 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-14T06:21:24.825 INFO:teuthology.task.internal:Creating archive directory... 2024-09-14T06:21:24.826 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T06:21:24.849 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T06:21:24.942 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-14T06:21:24.966 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-14T06:21:24.966 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T06:21:24.968 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T06:21:25.005 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T06:21:25.016 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T06:21:25.017 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T06:21:25.027 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T06:21:25.029 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-14T06:21:25.035 INFO:teuthology.task.internal:Configuring sudo... 2024-09-14T06:21:25.035 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T06:21:25.062 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T06:21:25.090 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-14T06:21:25.098 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-14T06:21:25.099 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T06:21:25.130 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T06:21:25.146 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T06:21:25.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T06:21:25.281 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T06:21:25.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T06:21:25.408 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:21:25.408 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T06:21:25.475 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T06:21:25.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T06:21:25.592 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T06:21:25.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T06:21:25.680 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:21:25.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T06:21:25.746 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-09-14T06:21:25.749 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-09-14T06:21:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T06:21:25.832 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T06:21:26.307 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-14T06:21:26.315 INFO:teuthology.task.internal:Starting timer... 2024-09-14T06:21:26.315 INFO:teuthology.run_tasks:Running task pcp... 2024-09-14T06:21:26.330 INFO:teuthology.run_tasks:Running task selinux... 2024-09-14T06:21:26.338 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-09-14T06:21:26.388 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-09-14T06:21:26.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-09-14T06:21:26.434 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-09-14T06:21:26.436 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-14T06:21:26.436 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-09-14T06:21:26.455 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-09-14T06:21:26.456 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-09-14T06:21:26.503 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-09-14T06:21:26.503 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-09-14T06:21:26.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T06:21:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:21:26.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T06:21:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:21:26.570 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-14T06:21:26.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-09-14T06:21:26.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-09-14T06:21:26.645 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-14T06:21:26.654 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-14T06:21:26.656 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-14T06:21:26.766 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-14T06:21:26.776 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-14T06:21:26.777 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-14T06:25:09.338 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-09-14T06:25:09.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-14T06:25:09.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:25:09.422 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-14T06:25:09.501 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-14T06:25:09.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-09-14T06:25:09.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T06:25:09.575 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-09-14T06:25:09.652 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-09-14T06:25:09.652 INFO:teuthology.run_tasks:Running task clock... 2024-09-14T06:25:09.666 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-14T06:25:09.666 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T06:25:09.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T06:25:09.669 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T06:25:09.669 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T06:25:09.698 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T06:25:09.713 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T06:25:09.722 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T06:25:09.738 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T06:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-09-14T06:25:09.751 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-09-14T06:25:09.763 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-09-14T06:25:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T06:25:09.774 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-09-14T06:25:09.781 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T06:25:09.787 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T06:25:09.801 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T06:25:09.838 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-09-14T06:25:09.841 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T06:25:09.841 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-09-14T06:25:09.841 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.841 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.842 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.842 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.855 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-09-14T06:25:09.859 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T06:25:09.859 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-09-14T06:25:09.859 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.859 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.859 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.860 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:25:09.860 INFO:teuthology.run_tasks:Running task install... 2024-09-14T06:25:09.869 DEBUG:teuthology.task.install:project ceph 2024-09-14T06:25:09.869 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}} 2024-09-14T06:25:09.869 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'} 2024-09-14T06:25:09.869 INFO:teuthology.task.install:Using flavor: default 2024-09-14T06:25:09.874 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-14T06:25:09.874 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-14T06:25:09.875 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'tag': None, 'wait_for_package': False} 2024-09-14T06:25:09.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T06:25:09.877 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'tag': None, 'wait_for_package': False} 2024-09-14T06:25:09.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T06:25:10.025 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/ 2024-09-14T06:25:10.025 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4775.g7f24e589 2024-09-14T06:25:10.031 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/ 2024-09-14T06:25:10.031 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4775.g7f24e589 2024-09-14T06:25:10.137 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T06:25:10.138 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:25:10.138 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T06:25:10.148 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T06:25:10.148 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:25:10.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T06:25:10.165 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-14T06:25:10.165 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f24e589328ae9e639c0009d93340e5b235d0a47/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T06:25:10.176 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-14T06:25:10.176 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f24e589328ae9e639c0009d93340e5b235d0a47/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T06:25:10.231 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T06:25:10.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T06:25:10.306 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T06:25:10.320 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T06:25:10.370 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-09-14T06:25:10.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-09-14T06:25:10.385 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-09-14T06:25:10.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-09-14T06:25:10.614 INFO:teuthology.orchestra.run.smithi170.stdout:45 files removed 2024-09-14T06:25:10.626 INFO:teuthology.orchestra.run.smithi001.stdout:45 files removed 2024-09-14T06:25:10.644 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2024-09-14T06:25:10.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw 2024-09-14T06:25:11.206 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 300 kB/s | 84 kB 00:00 2024-09-14T06:25:11.223 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 310 kB/s | 84 kB 00:00 2024-09-14T06:25:11.491 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-14T06:25:11.524 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-14T06:25:11.775 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-14T06:25:11.799 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-14T06:25:13.633 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-14T06:25:14.618 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.2 MB 00:02 2024-09-14T06:25:18.044 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 20 MB 00:03 2024-09-14T06:25:23.546 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-14T06:25:25.029 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 112 kB/s | 19 kB 00:00 2024-09-14T06:25:25.809 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-14T06:25:31.565 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-14T06:25:31.759 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 20 MB 00:16 2024-09-14T06:25:34.085 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 11 M 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.5 M 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-14T06:25:34.101 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 22 M 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 25 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 755 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 508 k 2024-09-14T06:25:34.102 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 45 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 143 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 161 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 321 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 306 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 99 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-14T06:25:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Install 31 Packages 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 2 Packages 2024-09-14T06:25:34.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:25:34.105 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 62 M 2024-09-14T06:25:34.105 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:25:35.024 INFO:teuthology.orchestra.run.smithi001.stdout:(1/33): ceph-base-19.3.0-4775.g7f24e589.el9.x86 9.9 MB/s | 5.5 MB 00:00 2024-09-14T06:25:35.099 INFO:teuthology.orchestra.run.smithi001.stdout:(2/33): ceph-selinux-19.3.0-4775.g7f24e589.el9. 337 kB/s | 25 kB 00:00 2024-09-14T06:25:35.199 INFO:teuthology.orchestra.run.smithi001.stdout:(3/33): libcephfs2-19.3.0-4775.g7f24e589.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-14T06:25:35.325 INFO:teuthology.orchestra.run.smithi001.stdout:(4/33): ceph-radosgw-19.3.0-4775.g7f24e589.el9. 13 MB/s | 11 MB 00:00 2024-09-14T06:25:35.383 INFO:teuthology.orchestra.run.smithi001.stdout:(5/33): libradosstriper1-19.3.0-4775.g7f24e589. 2.7 MB/s | 508 kB 00:00 2024-09-14T06:25:35.450 INFO:teuthology.orchestra.run.smithi001.stdout:(6/33): python3-ceph-argparse-19.3.0-4775.g7f24 677 kB/s | 45 kB 00:00 2024-09-14T06:25:35.526 INFO:teuthology.orchestra.run.smithi001.stdout:(7/33): python3-ceph-common-19.3.0-4775.g7f24e5 1.9 MB/s | 143 kB 00:00 2024-09-14T06:25:35.618 INFO:teuthology.orchestra.run.smithi001.stdout:(8/33): librgw2-19.3.0-4775.g7f24e589.el9.x86_6 19 MB/s | 5.5 MB 00:00 2024-09-14T06:25:35.643 INFO:teuthology.orchestra.run.smithi001.stdout:(9/33): python3-cephfs-19.3.0-4775.g7f24e589.el 1.4 MB/s | 161 kB 00:00 2024-09-14T06:25:35.668 INFO:teuthology.orchestra.run.smithi001.stdout:(10/33): python3-rados-19.3.0-4775.g7f24e589.el 6.3 MB/s | 321 kB 00:00 2024-09-14T06:25:35.693 INFO:teuthology.orchestra.run.smithi001.stdout:(11/33): python3-rbd-19.3.0-4775.g7f24e589.el9. 5.9 MB/s | 306 kB 00:00 2024-09-14T06:25:35.719 INFO:teuthology.orchestra.run.smithi001.stdout:(12/33): python3-rgw-19.3.0-4775.g7f24e589.el9. 1.9 MB/s | 99 kB 00:00 2024-09-14T06:25:35.927 INFO:teuthology.orchestra.run.smithi001.stdout:(13/33): ceph-common-19.3.0-4775.g7f24e589.el9. 15 MB/s | 22 MB 00:01 2024-09-14T06:25:35.986 INFO:teuthology.orchestra.run.smithi001.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-09-14T06:25:36.036 INFO:teuthology.orchestra.run.smithi001.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-09-14T06:25:36.061 INFO:teuthology.orchestra.run.smithi001.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 498 kB/s | 171 kB 00:00 2024-09-14T06:25:36.095 INFO:teuthology.orchestra.run.smithi001.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 160 kB 00:00 2024-09-14T06:25:36.137 INFO:teuthology.orchestra.run.smithi001.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-14T06:25:36.160 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-14T06:25:36.162 INFO:teuthology.orchestra.run.smithi001.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-14T06:25:36.188 INFO:teuthology.orchestra.run.smithi001.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 459 kB/s | 42 kB 00:00 2024-09-14T06:25:36.204 INFO:teuthology.orchestra.run.smithi001.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 861 kB/s | 36 kB 00:00 2024-09-14T06:25:36.230 INFO:teuthology.orchestra.run.smithi001.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-14T06:25:36.247 INFO:teuthology.orchestra.run.smithi001.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 806 kB/s | 47 kB 00:00 2024-09-14T06:25:36.305 INFO:teuthology.orchestra.run.smithi001.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-14T06:25:36.322 INFO:teuthology.orchestra.run.smithi001.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-14T06:25:36.464 INFO:teuthology.orchestra.run.smithi001.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 129 kB/s | 26 kB 00:00 2024-09-14T06:25:36.489 INFO:teuthology.orchestra.run.smithi001.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 404 kB/s | 67 kB 00:00 2024-09-14T06:25:36.523 INFO:teuthology.orchestra.run.smithi001.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-14T06:25:36.565 INFO:teuthology.orchestra.run.smithi001.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-14T06:25:36.624 INFO:teuthology.orchestra.run.smithi001.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-14T06:25:36.732 INFO:teuthology.orchestra.run.smithi001.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-09-14T06:25:36.808 INFO:teuthology.orchestra.run.smithi001.stdout:(32/33): librados2-19.3.0-4775.g7f24e589.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-14T06:25:36.899 INFO:teuthology.orchestra.run.smithi001.stdout:(33/33): librbd1-19.3.0-4775.g7f24e589.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-14T06:25:36.905 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:25:36.905 INFO:teuthology.orchestra.run.smithi001.stdout:Total 22 MB/s | 62 MB 00:02 2024-09-14T06:25:37.027 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:25:37.067 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:25:37.068 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:25:37.763 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:25:37.763 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:25:37.844 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-14T06:25:38.665 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-14T06:25:39.332 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:25:39.410 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-14T06:25:39.438 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-14T06:25:39.471 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-14T06:25:39.494 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T06:25:39.517 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T06:25:39.664 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T06:25:39.718 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T06:25:40.133 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-14T06:25:40.214 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T06:25:40.291 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T06:25:40.381 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 8/35 2024-09-14T06:25:40.408 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T06:25:40.482 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T06:25:40.516 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-14T06:25:40.561 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 11/35 2024-09-14T06:25:40.631 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T06:25:40.660 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T06:25:40.725 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T06:25:40.759 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-14T06:25:40.879 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-14T06:25:40.926 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-14T06:25:40.981 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-14T06:25:41.025 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-14T06:25:41.054 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-14T06:25:41.087 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T06:25:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-14T06:25:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-14T06:25:41.517 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T06:25:41.620 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T06:25:41.658 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 24/35 2024-09-14T06:25:41.700 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-14T06:25:42.231 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-14T06:25:42.261 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T06:25:42.333 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T06:25:42.414 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 28/35 2024-09-14T06:25:42.455 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 29/35 2024-09-14T06:25:44.071 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-14T06:25:44.882 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T06:25:44.927 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T06:25:45.543 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T06:25:45.580 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T06:25:45.661 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T06:25:45.661 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T06:25:45.661 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T06:25:45.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:25:45.690 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T06:25:46.533 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:25:46.557 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 11 M 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.5 M 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 22 M 2024-09-14T06:25:46.558 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 25 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 755 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 508 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-14T06:25:46.559 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 45 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 143 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 161 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 321 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 306 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 99 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-14T06:25:46.560 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:25:46.561 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:25:46.561 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:25:46.561 INFO:teuthology.orchestra.run.smithi170.stdout:Install 31 Packages 2024-09-14T06:25:46.561 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2024-09-14T06:25:46.561 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:25:46.561 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 62 M 2024-09-14T06:25:46.561 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:25:47.398 INFO:teuthology.orchestra.run.smithi170.stdout:(1/33): ceph-base-19.3.0-4775.g7f24e589.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-14T06:25:47.464 INFO:teuthology.orchestra.run.smithi170.stdout:(2/33): ceph-selinux-19.3.0-4775.g7f24e589.el9. 379 kB/s | 25 kB 00:00 2024-09-14T06:25:47.615 INFO:teuthology.orchestra.run.smithi170.stdout:(3/33): ceph-radosgw-19.3.0-4775.g7f24e589.el9. 14 MB/s | 11 MB 00:00 2024-09-14T06:25:47.648 INFO:teuthology.orchestra.run.smithi170.stdout:(4/33): libcephfs2-19.3.0-4775.g7f24e589.el9.x8 4.0 MB/s | 755 kB 00:00 2024-09-14T06:25:47.673 INFO:teuthology.orchestra.run.smithi170.stdout:(5/33): libradosstriper1-19.3.0-4775.g7f24e589. 8.5 MB/s | 508 kB 00:00 2024-09-14T06:25:47.724 INFO:teuthology.orchestra.run.smithi170.stdout:(6/33): python3-ceph-argparse-19.3.0-4775.g7f24 903 kB/s | 45 kB 00:00 2024-09-14T06:25:47.816 INFO:teuthology.orchestra.run.smithi170.stdout:(7/33): python3-ceph-common-19.3.0-4775.g7f24e5 1.5 MB/s | 143 kB 00:00 2024-09-14T06:25:47.899 INFO:teuthology.orchestra.run.smithi170.stdout:(8/33): librgw2-19.3.0-4775.g7f24e589.el9.x86_6 22 MB/s | 5.5 MB 00:00 2024-09-14T06:25:47.924 INFO:teuthology.orchestra.run.smithi170.stdout:(9/33): python3-cephfs-19.3.0-4775.g7f24e589.el 1.5 MB/s | 161 kB 00:00 2024-09-14T06:25:47.958 INFO:teuthology.orchestra.run.smithi170.stdout:(10/33): python3-rados-19.3.0-4775.g7f24e589.el 5.4 MB/s | 321 kB 00:00 2024-09-14T06:25:47.983 INFO:teuthology.orchestra.run.smithi170.stdout:(11/33): python3-rbd-19.3.0-4775.g7f24e589.el9. 5.1 MB/s | 306 kB 00:00 2024-09-14T06:25:48.009 INFO:teuthology.orchestra.run.smithi170.stdout:(12/33): python3-rgw-19.3.0-4775.g7f24e589.el9. 1.9 MB/s | 99 kB 00:00 2024-09-14T06:25:48.226 INFO:teuthology.orchestra.run.smithi170.stdout:(13/33): ceph-common-19.3.0-4775.g7f24e589.el9. 16 MB/s | 22 MB 00:01 2024-09-14T06:25:48.826 INFO:teuthology.orchestra.run.smithi170.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 127 kB/s | 107 kB 00:00 2024-09-14T06:25:48.868 INFO:teuthology.orchestra.run.smithi170.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 199 kB/s | 171 kB 00:00 2024-09-14T06:25:48.951 INFO:teuthology.orchestra.run.smithi170.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 221 kB/s | 160 kB 00:00 2024-09-14T06:25:49.410 INFO:teuthology.orchestra.run.smithi170.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 77 kB/s | 45 kB 00:00 2024-09-14T06:25:49.744 INFO:teuthology.orchestra.run.smithi170.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 369 kB/s | 292 kB 00:00 2024-09-14T06:25:49.769 INFO:teuthology.orchestra.run.smithi170.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 735 kB/s | 662 kB 00:00 2024-09-14T06:25:49.978 INFO:teuthology.orchestra.run.smithi170.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 74 kB/s | 42 kB 00:00 2024-09-14T06:25:50.336 INFO:teuthology.orchestra.run.smithi170.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 63 kB/s | 36 kB 00:00 2024-09-14T06:25:50.395 INFO:teuthology.orchestra.run.smithi170.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 5.2 MB/s | 308 kB 00:00 2024-09-14T06:25:50.503 INFO:teuthology.orchestra.run.smithi170.stdout:(23/33): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-14T06:25:50.537 INFO:teuthology.orchestra.run.smithi170.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 778 kB/s | 26 kB 00:00 2024-09-14T06:25:50.563 INFO:teuthology.orchestra.run.smithi170.stdout:(25/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 83 kB/s | 47 kB 00:00 2024-09-14T06:25:50.588 INFO:teuthology.orchestra.run.smithi170.stdout:(26/33): qatlib-24.02.0-1.el9.x86_64.rpm 262 kB/s | 221 kB 00:00 2024-09-14T06:25:50.621 INFO:teuthology.orchestra.run.smithi170.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 579 kB/s | 49 kB 00:00 2024-09-14T06:25:50.655 INFO:teuthology.orchestra.run.smithi170.stdout:(28/33): libunwind-1.6.2-1.el9.x86_64.rpm 742 kB/s | 67 kB 00:00 2024-09-14T06:25:50.680 INFO:teuthology.orchestra.run.smithi170.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-14T06:25:50.731 INFO:teuthology.orchestra.run.smithi170.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 22 MB/s | 1.6 MB 00:00 2024-09-14T06:25:50.764 INFO:teuthology.orchestra.run.smithi170.stdout:(31/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 839 kB 00:00 2024-09-14T06:25:50.998 INFO:teuthology.orchestra.run.smithi170.stdout:(32/33): librbd1-19.3.0-4775.g7f24e589.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-14T06:25:51.056 INFO:teuthology.orchestra.run.smithi170.stdout:(33/33): librados2-19.3.0-4775.g7f24e589.el9.x8 9.2 MB/s | 3.5 MB 00:00 2024-09-14T06:25:51.063 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:25:51.063 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 62 MB 00:04 2024-09-14T06:25:51.174 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:25:51.211 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:25:51.211 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:25:51.897 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:25:51.898 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:25:53.411 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:25:53.500 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-14T06:25:53.528 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-14T06:25:53.553 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-14T06:25:53.576 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T06:25:53.597 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T06:25:53.796 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T06:25:53.857 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T06:25:54.233 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-14T06:25:54.305 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T06:25:54.398 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T06:25:54.497 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 8/35 2024-09-14T06:25:54.532 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T06:25:54.595 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T06:25:54.624 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-14T06:25:54.669 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 11/35 2024-09-14T06:25:54.738 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T06:25:54.759 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T06:25:54.824 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T06:25:54.858 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-14T06:25:54.977 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-14T06:25:55.009 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-14T06:25:55.059 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-14T06:25:55.091 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-14T06:25:55.120 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-14T06:25:55.153 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T06:25:55.191 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-14T06:25:55.518 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-14T06:25:55.566 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T06:25:55.664 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T06:25:55.704 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 24/35 2024-09-14T06:25:55.748 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-14T06:25:56.208 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T06:25:56.208 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-09-14T06:25:56.208 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-09-14T06:25:56.208 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-09-14T06:25:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-09-14T06:25:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-09-14T06:25:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-09-14T06:25:56.209 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-09-14T06:25:56.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:25:56.251 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-14T06:25:56.277 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T06:25:56.349 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T06:25:56.427 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 28/35 2024-09-14T06:25:56.455 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 29/35 2024-09-14T06:25:57.290 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T06:25:57.317 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T06:25:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:25:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T06:25:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T06:25:57.317 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T06:25:57.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:25:57.319 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T06:25:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T06:25:57.409 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T06:25:58.607 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 1/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 3/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 4/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 5/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 6/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 8/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 9/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 10/35 2024-09-14T06:25:58.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 11/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 13/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-14T06:25:58.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-14T06:25:58.610 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 34/35 2024-09-14T06:25:58.813 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T06:25:58.843 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-14T06:25:59.401 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.402 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-14T06:25:59.403 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-14T06:25:59.404 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-20.el9.x86_64 2024-09-14T06:25:59.404 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T06:25:59.404 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:25:59.404 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:25:59.417 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T06:25:59.458 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T06:25:59.550 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T06:25:59.550 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T06:25:59.550 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T06:25:59.550 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:25:59.584 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T06:25:59.746 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test 2024-09-14T06:26:00.328 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:29 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:00.466 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:00.466 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:00.466 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:26:00.466 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:00.466 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 50 M 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 162 k 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-09-14T06:26:00.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:00.468 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 51 M 2024-09-14T06:26:00.468 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 217 M 2024-09-14T06:26:00.468 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:00.648 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-14T06:26:00.689 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-14T06:26:00.740 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): libcephsqlite-19.3.0-4775.g7f24e589.el9. 729 kB/s | 162 kB 00:00 2024-09-14T06:26:02.198 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): ceph-test-19.3.0-4775.g7f24e589.el9.x86_ 30 MB/s | 50 MB 00:01 2024-09-14T06:26:02.200 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:02.201 INFO:teuthology.orchestra.run.smithi001.stdout:Total 29 MB/s | 51 MB 00:01 2024-09-14T06:26:02.229 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:26:02.252 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:26:02.252 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:26:02.685 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:26:02.686 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:26:03.252 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:26:03.342 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T06:26:03.382 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T06:26:03.408 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T06:26:08.439 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T06:26:08.527 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-09-14T06:26:09.624 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:10.619 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T06:26:10.644 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T06:26:10.644 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:10.644 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T06:26:10.644 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T06:26:10.644 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T06:26:10.644 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:10.646 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T06:26:10.729 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T06:26:10.729 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T06:26:11.099 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T06:26:11.099 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 1/4 2024-09-14T06:26:11.099 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2/4 2024-09-14T06:26:11.099 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:11.420 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:11.689 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 1/35 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2/35 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 3/35 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 4/35 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 5/35 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 6/35 2024-09-14T06:26:11.946 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 8/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 9/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 10/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 11/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 13/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T06:26:11.947 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-14T06:26:11.948 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-14T06:26:11.949 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-14T06:26:11.949 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-14T06:26:11.949 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T06:26:11.949 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-14T06:26:11.949 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 34/35 2024-09-14T06:26:12.273 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:41 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:12.422 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:12.424 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:12.424 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 6.6 k 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 2.4 M 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 1.5 M 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 291 k 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 4.8 M 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 17 M 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-14T06:26:12.425 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T06:26:12.426 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T06:26:12.427 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout:Install 37 Packages 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:12.428 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 31 M 2024-09-14T06:26:12.429 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 109 M 2024-09-14T06:26:12.429 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:13.032 INFO:teuthology.orchestra.run.smithi001.stdout:(1/37): ceph-19.3.0-4775.g7f24e589.el9.x86_64.r 46 kB/s | 6.6 kB 00:00 2024-09-14T06:26:13.053 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T06:26:13.053 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:13.053 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-09-14T06:26:13.053 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.053 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.053 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T06:26:13.054 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-14T06:26:13.055 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-14T06:26:13.056 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-14T06:26:13.056 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-20.el9.x86_64 2024-09-14T06:26:13.056 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T06:26:13.056 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:13.056 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:13.199 INFO:teuthology.orchestra.run.smithi001.stdout:(2/37): ceph-mgr-19.3.0-4775.g7f24e589.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-14T06:26:13.258 INFO:teuthology.orchestra.run.smithi001.stdout:(3/37): ceph-mds-19.3.0-4775.g7f24e589.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-14T06:26:13.350 INFO:teuthology.orchestra.run.smithi001.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4775.g7f24 3.1 MB/s | 291 kB 00:00 2024-09-14T06:26:13.416 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2024-09-14T06:26:13.500 INFO:teuthology.orchestra.run.smithi001.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-14T06:26:13.659 INFO:teuthology.orchestra.run.smithi001.stdout:(6/37): ceph-mon-19.3.0-4775.g7f24e589.el9.x86_ 7.6 MB/s | 4.8 MB 00:00 2024-09-14T06:26:13.826 INFO:teuthology.orchestra.run.smithi001.stdout:(7/37): ceph-osd-19.3.0-4775.g7f24e589.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-14T06:26:13.859 INFO:teuthology.orchestra.run.smithi001.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 3.4 MB/s | 1.2 MB 00:00 2024-09-14T06:26:13.918 INFO:teuthology.orchestra.run.smithi001.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-14T06:26:13.945 INFO:teuthology.orchestra.run.smithi001.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 375 kB/s | 106 kB 00:00 2024-09-14T06:26:13.968 INFO:teuthology.orchestra.run.smithi001.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 952 kB/s | 135 kB 00:00 2024-09-14T06:26:13.987 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:29 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:14.002 INFO:teuthology.orchestra.run.smithi001.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-14T06:26:14.080 INFO:teuthology.orchestra.run.smithi001.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 444 kB/s | 35 kB 00:00 2024-09-14T06:26:14.114 INFO:teuthology.orchestra.run.smithi001.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-14T06:26:14.128 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 50 M 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 162 k 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T06:26:14.129 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:14.130 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:26:14.130 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:14.130 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-09-14T06:26:14.130 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:14.130 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 51 M 2024-09-14T06:26:14.130 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 217 M 2024-09-14T06:26:14.131 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:26:14.139 INFO:teuthology.orchestra.run.smithi001.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 963 kB/s | 188 kB 00:00 2024-09-14T06:26:14.165 INFO:teuthology.orchestra.run.smithi001.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 921 kB/s | 77 kB 00:00 2024-09-14T06:26:14.181 INFO:teuthology.orchestra.run.smithi001.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 760 kB/s | 51 kB 00:00 2024-09-14T06:26:14.198 INFO:teuthology.orchestra.run.smithi001.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-09-14T06:26:14.240 INFO:teuthology.orchestra.run.smithi001.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-14T06:26:14.265 INFO:teuthology.orchestra.run.smithi001.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 650 kB/s | 43 kB 00:00 2024-09-14T06:26:14.291 INFO:teuthology.orchestra.run.smithi001.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-14T06:26:14.316 INFO:teuthology.orchestra.run.smithi001.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.3 MB/s | 172 kB 00:00 2024-09-14T06:26:14.319 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-14T06:26:14.350 INFO:teuthology.orchestra.run.smithi001.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 320 kB/s | 11 kB 00:00 2024-09-14T06:26:14.361 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-14T06:26:14.375 INFO:teuthology.orchestra.run.smithi001.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.0 MB/s | 23 kB 00:00 2024-09-14T06:26:14.400 INFO:teuthology.orchestra.run.smithi001.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-09-14T06:26:14.403 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): libcephsqlite-19.3.0-4775.g7f24e589.el9. 646 kB/s | 162 kB 00:00 2024-09-14T06:26:14.426 INFO:teuthology.orchestra.run.smithi001.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-14T06:26:14.451 INFO:teuthology.orchestra.run.smithi001.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-14T06:26:14.476 INFO:teuthology.orchestra.run.smithi001.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-14T06:26:14.501 INFO:teuthology.orchestra.run.smithi001.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-14T06:26:14.527 INFO:teuthology.orchestra.run.smithi001.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-14T06:26:14.552 INFO:teuthology.orchestra.run.smithi001.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-14T06:26:14.577 INFO:teuthology.orchestra.run.smithi001.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-14T06:26:14.602 INFO:teuthology.orchestra.run.smithi001.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-14T06:26:14.628 INFO:teuthology.orchestra.run.smithi001.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-14T06:26:14.653 INFO:teuthology.orchestra.run.smithi001.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-14T06:26:14.678 INFO:teuthology.orchestra.run.smithi001.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-14T06:26:14.703 INFO:teuthology.orchestra.run.smithi001.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-14T06:26:14.707 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:14.707 INFO:teuthology.orchestra.run.smithi001.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-14T06:26:14.862 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:26:14.906 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:26:14.906 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:26:15.377 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:26:15.377 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:26:15.711 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): ceph-test-19.3.0-4775.g7f24e589.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-14T06:26:15.714 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:15.714 INFO:teuthology.orchestra.run.smithi170.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-14T06:26:15.742 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:26:15.762 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:26:15.763 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:26:16.160 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:26:16.161 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:26:16.721 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:26:16.758 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:26:16.820 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T06:26:16.840 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-14T06:26:16.873 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-14T06:26:16.877 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T06:26:16.900 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-14T06:26:16.914 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T06:26:16.976 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-14T06:26:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-14T06:26:17.057 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-14T06:26:17.153 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-14T06:26:17.247 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-14T06:26:17.294 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-14T06:26:17.329 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-14T06:26:17.358 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-14T06:26:17.393 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-14T06:26:17.426 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-14T06:26:17.459 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-14T06:26:17.502 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-14T06:26:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-14T06:26:17.586 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-14T06:26:17.625 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-14T06:26:17.701 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-14T06:26:17.862 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-14T06:26:17.945 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-14T06:26:17.996 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-14T06:26:18.034 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-14T06:26:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-14T06:26:18.157 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-14T06:26:18.361 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-14T06:26:18.413 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-14T06:26:18.495 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-14T06:26:18.591 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-14T06:26:18.637 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-14T06:26:18.773 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-14T06:26:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 32/37 2024-09-14T06:26:18.979 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T06:26:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T06:26:19.007 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:19.007 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T06:26:19.007 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T06:26:19.007 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T06:26:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:20.564 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T06:26:20.593 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T06:26:20.593 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:20.593 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T06:26:20.593 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T06:26:20.593 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T06:26:20.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:21.081 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T06:26:21.107 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T06:26:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T06:26:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T06:26:21.107 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T06:26:21.107 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:21.349 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T06:26:21.373 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T06:26:21.374 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:21.374 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T06:26:21.374 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T06:26:21.374 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T06:26:21.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:21.440 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T06:26:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T06:26:21.786 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 1/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 3/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 4/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 5/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 6/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-14T06:26:23.341 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-14T06:26:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-14T06:26:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-14T06:26:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-14T06:26:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-14T06:26:23.344 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-14T06:26:23.344 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-14T06:26:23.344 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-14T06:26:23.344 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-14T06:26:23.344 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-14T06:26:23.344 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-14T06:26:23.344 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T06:26:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T06:26:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T06:26:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T06:26:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T06:26:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T06:26:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:24.281 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T06:26:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 1/4 2024-09-14T06:26:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2/4 2024-09-14T06:26:24.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T06:26:24.585 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:24.586 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:24.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base 2024-09-14T06:26:24.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2024-09-14T06:26:25.205 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:25.300 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:26:25.345 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:25.346 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:26:25.346 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:25.413 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install cephadm 2024-09-14T06:26:25.457 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:41 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:25.604 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:25.607 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:25.607 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:26:25.607 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:25.607 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:26:25.607 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 6.6 k 2024-09-14T06:26:25.607 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 2.4 M 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 1.5 M 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 291 k 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 4.8 M 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 17 M 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T06:26:25.608 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T06:26:25.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T06:26:25.610 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T06:26:25.611 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T06:26:25.611 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:25.611 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:26:25.611 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:25.611 INFO:teuthology.orchestra.run.smithi170.stdout:Install 37 Packages 2024-09-14T06:26:25.611 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:25.611 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 31 M 2024-09-14T06:26:25.612 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 109 M 2024-09-14T06:26:25.612 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:26:25.995 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:26.133 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:26.133 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:26.133 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 783 k 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 783 k 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 789 k 2024-09-14T06:26:26.134 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:26.234 INFO:teuthology.orchestra.run.smithi170.stdout:(1/37): ceph-19.3.0-4775.g7f24e589.el9.x86_64.r 51 kB/s | 6.6 kB 00:00 2024-09-14T06:26:26.431 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm-19.3.0-4775.g7f24e589.el9.noarch.rpm 2.6 MB/s | 783 kB 00:00 2024-09-14T06:26:26.431 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:26.432 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.6 MB/s | 783 kB 00:00 2024-09-14T06:26:26.432 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:26:26.437 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:26:26.437 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:26:26.443 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:26:26.443 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:26:26.451 INFO:teuthology.orchestra.run.smithi170.stdout:(2/37): ceph-mds-19.3.0-4775.g7f24e589.el9.x86_ 6.9 MB/s | 2.4 MB 00:00 2024-09-14T06:26:26.485 INFO:teuthology.orchestra.run.smithi170.stdout:(3/37): ceph-mgr-19.3.0-4775.g7f24e589.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-09-14T06:26:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:26:26.577 INFO:teuthology.orchestra.run.smithi170.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4775.g7f24 3.1 MB/s | 291 kB 00:00 2024-09-14T06:26:26.719 INFO:teuthology.orchestra.run.smithi170.stdout:(5/37): ceph-mon-19.3.0-4775.g7f24e589.el9.x86_ 9.9 MB/s | 4.8 MB 00:00 2024-09-14T06:26:26.744 INFO:teuthology.orchestra.run.smithi170.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-14T06:26:26.786 INFO:teuthology.orchestra.run.smithi170.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-14T06:26:26.828 INFO:teuthology.orchestra.run.smithi170.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-14T06:26:26.870 INFO:teuthology.orchestra.run.smithi170.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-14T06:26:26.906 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:26.994 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:27.037 INFO:teuthology.orchestra.run.smithi170.stdout:(10/37): ceph-osd-19.3.0-4775.g7f24e589.el9.x86 29 MB/s | 17 MB 00:00 2024-09-14T06:26:27.062 INFO:teuthology.orchestra.run.smithi170.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.1 MB/s | 215 kB 00:00 2024-09-14T06:26:27.104 INFO:teuthology.orchestra.run.smithi170.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 3.2 MB/s | 1.2 MB 00:00 2024-09-14T06:26:27.171 INFO:teuthology.orchestra.run.smithi170.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 520 kB/s | 35 kB 00:00 2024-09-14T06:26:27.196 INFO:teuthology.orchestra.run.smithi170.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-14T06:26:27.222 INFO:teuthology.orchestra.run.smithi170.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-14T06:26:27.247 INFO:teuthology.orchestra.run.smithi170.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-14T06:26:27.314 INFO:teuthology.orchestra.run.smithi170.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 434 kB/s | 51 kB 00:00 2024-09-14T06:26:27.339 INFO:teuthology.orchestra.run.smithi170.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 355 kB/s | 42 kB 00:00 2024-09-14T06:26:27.364 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:27.364 INFO:teuthology.orchestra.run.smithi170.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-09-14T06:26:27.423 INFO:teuthology.orchestra.run.smithi170.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-09-14T06:26:27.448 INFO:teuthology.orchestra.run.smithi170.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 520 kB/s | 43 kB 00:00 2024-09-14T06:26:27.475 INFO:teuthology.orchestra.run.smithi170.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 413 kB/s | 11 kB 00:00 2024-09-14T06:26:27.499 INFO:teuthology.orchestra.run.smithi170.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-14T06:26:27.525 INFO:teuthology.orchestra.run.smithi170.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 361 kB/s | 18 kB 00:00 2024-09-14T06:26:27.549 INFO:teuthology.orchestra.run.smithi170.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-14T06:26:27.575 INFO:teuthology.orchestra.run.smithi170.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-14T06:26:27.600 INFO:teuthology.orchestra.run.smithi170.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-14T06:26:27.613 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:27.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:27.613 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:26:27.614 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:27.614 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:27.614 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:27.634 INFO:teuthology.orchestra.run.smithi170.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-14T06:26:27.659 INFO:teuthology.orchestra.run.smithi170.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-09-14T06:26:27.684 INFO:teuthology.orchestra.run.smithi170.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-09-14T06:26:27.710 INFO:teuthology.orchestra.run.smithi170.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-14T06:26:27.735 INFO:teuthology.orchestra.run.smithi170.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-14T06:26:27.760 INFO:teuthology.orchestra.run.smithi170.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-14T06:26:27.794 INFO:teuthology.orchestra.run.smithi170.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-14T06:26:27.828 INFO:teuthology.orchestra.run.smithi170.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 597 kB/s | 20 kB 00:00 2024-09-14T06:26:27.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T06:26:27.861 INFO:teuthology.orchestra.run.smithi170.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-14T06:26:27.886 INFO:teuthology.orchestra.run.smithi170.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-09-14T06:26:27.890 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:27.890 INFO:teuthology.orchestra.run.smithi170.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-14T06:26:28.033 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:26:28.072 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:26:28.073 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:26:28.434 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:57 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:28.574 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:28.575 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:28.575 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-14T06:26:28.575 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:28.575 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:28.575 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 145 k 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 145 k 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 432 k 2024-09-14T06:26:28.576 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:28.585 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:26:28.585 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:26:28.788 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-19.3.0-4775.g7f24e5 683 kB/s | 145 kB 00:00 2024-09-14T06:26:28.789 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:28.789 INFO:teuthology.orchestra.run.smithi001.stdout:Total 677 kB/s | 145 kB 00:00 2024-09-14T06:26:28.789 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:26:28.797 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:26:28.798 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:26:28.859 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:26:28.859 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:26:29.012 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:26:29.073 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T06:26:29.099 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T06:26:29.099 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:29.099 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T06:26:29.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:29.928 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:26:29.981 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-14T06:26:30.014 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-14T06:26:30.019 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T06:26:30.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:30.019 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:26:30.019 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:30.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:30.019 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:30.077 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-14T06:26:30.157 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-14T06:26:30.210 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-14T06:26:30.239 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr 2024-09-14T06:26:30.242 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-14T06:26:30.334 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-14T06:26:30.413 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-14T06:26:30.454 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-14T06:26:30.497 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-14T06:26:30.536 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-14T06:26:30.570 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-14T06:26:30.603 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-14T06:26:30.652 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-14T06:26:30.687 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-14T06:26:30.722 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-14T06:26:30.770 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-14T06:26:30.802 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-14T06:26:30.833 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:59 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:30.875 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-14T06:26:30.927 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:26:30.972 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:30.973 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:26:30.973 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:31.015 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-14T06:26:31.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T06:26:31.094 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-14T06:26:31.156 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-14T06:26:31.194 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-14T06:26:31.251 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-14T06:26:31.307 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-14T06:26:31.491 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-14T06:26:31.540 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-14T06:26:31.618 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-14T06:26:31.618 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:00 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:31.708 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-14T06:26:31.756 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-14T06:26:31.757 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:31.758 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:31.758 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:26:31.758 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:31.758 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:31.758 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 14 M 2024-09-14T06:26:31.758 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 34 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 16 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T06:26:31.759 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T06:26:31.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:31.760 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:31.760 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:31.760 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-09-14T06:26:31.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:31.760 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 19 M 2024-09-14T06:26:31.760 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 159 M 2024-09-14T06:26:31.761 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:31.885 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-14T06:26:32.066 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 32/37 2024-09-14T06:26:32.100 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4775.g7f2 121 kB/s | 16 kB 00:00 2024-09-14T06:26:32.101 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T06:26:32.117 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4775.g7f 223 kB/s | 34 kB 00:00 2024-09-14T06:26:32.126 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T06:26:32.126 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:32.127 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T06:26:32.127 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T06:26:32.127 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T06:26:32.127 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:32.317 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-14T06:26:32.459 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-09-14T06:26:32.626 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4775.g7f24e58 21 MB/s | 14 MB 00:00 2024-09-14T06:26:32.668 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-14T06:26:32.710 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 239 kB/s | 19 kB 00:00 2024-09-14T06:26:32.744 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.5 MB/s | 144 kB 00:00 2024-09-14T06:26:32.769 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-14T06:26:32.794 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-09-14T06:26:32.811 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-14T06:26:32.870 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-14T06:26:32.874 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:32.874 INFO:teuthology.orchestra.run.smithi001.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-14T06:26:32.922 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:26:32.942 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:26:32.942 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:26:33.223 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:26:33.224 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:26:33.593 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T06:26:33.620 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T06:26:33.620 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:33.620 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T06:26:33.620 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T06:26:33.620 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T06:26:33.620 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:33.848 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:26:33.921 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-14T06:26:33.965 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-14T06:26:34.012 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-14T06:26:34.041 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-14T06:26:34.068 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T06:26:34.092 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T06:26:34.093 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:34.093 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T06:26:34.093 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T06:26:34.093 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T06:26:34.093 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:34.149 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-14T06:26:34.281 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-14T06:26:34.328 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T06:26:34.351 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T06:26:34.351 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:34.351 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T06:26:34.351 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T06:26:34.351 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T06:26:34.352 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:34.381 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-14T06:26:34.417 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T06:26:34.430 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-14T06:26:34.457 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-14T06:26:34.490 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 10/12 2024-09-14T06:26:36.221 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 1/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 3/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 4/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 5/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 6/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-14T06:26:36.222 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-14T06:26:36.223 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-14T06:26:36.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-14T06:26:36.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-14T06:26:36.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-14T06:26:36.224 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-14T06:26:36.225 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:37.132 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T06:26:37.133 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T06:26:37.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:37.135 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:37.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2024-09-14T06:26:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 11/12 2024-09-14T06:26:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T06:26:38.084 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:38.173 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:26:38.214 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:38.215 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:26:38.215 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:38.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2024-09-14T06:26:38.857 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:38.989 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:38.990 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:38.990 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:26:38.990 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 783 k 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 783 k 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 789 k 2024-09-14T06:26:38.991 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:26:39.237 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-19.3.0-4775.g7f24e589.el9.noarch.rpm 3.1 MB/s | 783 kB 00:00 2024-09-14T06:26:39.237 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:39.237 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.1 MB/s | 783 kB 00:00 2024-09-14T06:26:39.238 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:26:39.243 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:26:39.243 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:26:39.249 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:26:39.249 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:26:39.342 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:26:39.823 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 1/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 2/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 3/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-14T06:26:39.841 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-14T06:26:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-14T06:26:40.171 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:40.172 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:40.303 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:40.419 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T06:26:40.557 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:26:40.557 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:40.557 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:26:40.557 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:40.557 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:40.557 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:40.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T06:26:41.012 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:10 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:41.154 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 7.4 M 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T06:26:41.156 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-09-14T06:26:41.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:41.158 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 38 M 2024-09-14T06:26:41.158 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 212 M 2024-09-14T06:26:41.158 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:41.347 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:57 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:41.487 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:41.487 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:41.487 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 145 k 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 145 k 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 432 k 2024-09-14T06:26:41.488 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:26:41.695 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-19.3.0-4775.g7f24e5 700 kB/s | 145 kB 00:00 2024-09-14T06:26:41.696 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:41.696 INFO:teuthology.orchestra.run.smithi170.stdout:Total 695 kB/s | 145 kB 00:00 2024-09-14T06:26:41.696 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:26:41.703 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:26:41.704 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:26:41.742 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 907 kB/s | 191 kB 00:00 2024-09-14T06:26:41.760 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:26:41.761 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:26:41.792 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-14T06:26:41.924 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:26:41.996 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T06:26:42.021 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T06:26:42.022 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:26:42.022 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T06:26:42.022 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:42.093 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-47 13 MB/s | 7.4 MB 00:00 2024-09-14T06:26:42.118 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 87 kB/s | 33 kB 00:00 2024-09-14T06:26:42.160 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 274 kB/s | 18 kB 00:00 2024-09-14T06:26:42.277 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 242 kB/s | 38 kB 00:00 2024-09-14T06:26:42.544 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 917 kB/s | 245 kB 00:00 2024-09-14T06:26:42.628 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-09-14T06:26:42.686 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-14T06:26:42.829 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-09-14T06:26:42.981 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T06:26:42.981 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:42.981 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:26:42.981 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:26:42.981 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:42.981 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:43.087 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-14T06:26:43.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2024-09-14T06:26:43.204 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 2.1 MB/s | 3.0 MB 00:01 2024-09-14T06:26:43.205 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:43.205 INFO:teuthology.orchestra.run.smithi001.stdout:Total 19 MB/s | 38 MB 00:02 2024-09-14T06:26:43.570 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:26:43.597 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:26:43.597 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:26:43.783 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:59 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:43.877 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:26:43.919 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:43.920 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:26:43.921 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:43.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T06:26:44.068 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:26:44.068 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:26:44.557 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:00 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:44.690 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:44.691 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:26:44.691 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:44.691 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:26:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 14 M 2024-09-14T06:26:44.691 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:26:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 34 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 16 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:26:44.692 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:26:44.693 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2024-09-14T06:26:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:44.693 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 19 M 2024-09-14T06:26:44.693 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 159 M 2024-09-14T06:26:44.693 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:26:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:26:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T06:26:44.952 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T06:26:45.066 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T06:26:45.104 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T06:26:45.108 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4775.g7f2 103 kB/s | 16 kB 00:00 2024-09-14T06:26:45.134 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4775.g7f 184 kB/s | 34 kB 00:00 2024-09-14T06:26:45.296 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T06:26:45.336 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T06:26:45.359 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-14T06:26:45.415 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T06:26:45.568 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): ceph-mgr-dashboard-19.3.0-4775.g7f24e58 23 MB/s | 14 MB 00:00 2024-09-14T06:26:45.618 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-14T06:26:45.643 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 777 kB/s | 19 kB 00:00 2024-09-14T06:26:45.710 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-14T06:26:45.735 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-14T06:26:45.761 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-14T06:26:45.786 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-14T06:26:45.819 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.8 MB/s | 863 kB 00:00 2024-09-14T06:26:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T06:26:46.237 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T06:26:47.548 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T06:26:48.061 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 350 kB/s | 1.0 MB 00:02 2024-09-14T06:26:48.065 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:48.065 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.7 MB/s | 19 MB 00:03 2024-09-14T06:26:48.120 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:26:48.138 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:26:48.138 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:26:48.414 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:26:48.414 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:26:48.630 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T06:26:48.659 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T06:26:49.038 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:26:49.129 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-14T06:26:49.172 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-14T06:26:49.219 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-14T06:26:49.258 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-14T06:26:49.360 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-14T06:26:49.493 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-14T06:26:49.594 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-14T06:26:49.646 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-14T06:26:49.674 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-14T06:26:49.707 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 10/12 2024-09-14T06:26:51.214 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 1/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T06:26:51.215 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T06:26:51.612 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:51.613 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:51.990 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-rook 2024-09-14T06:26:52.588 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:21 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:52.727 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 49 k 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T06:26:52.729 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout:Install 13 Packages 2024-09-14T06:26:52.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:52.731 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.2 M 2024-09-14T06:26:52.731 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-09-14T06:26:52.731 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:52.934 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 11/12 2024-09-14T06:26:52.959 INFO:teuthology.orchestra.run.smithi001.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 323 kB/s | 26 kB 00:00 2024-09-14T06:26:52.984 INFO:teuthology.orchestra.run.smithi001.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 183 kB/s | 19 kB 00:00 2024-09-14T06:26:52.994 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T06:26:53.060 INFO:teuthology.orchestra.run.smithi001.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-14T06:26:53.085 INFO:teuthology.orchestra.run.smithi001.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-14T06:26:53.110 INFO:teuthology.orchestra.run.smithi001.stdout:(5/13): ceph-mgr-rook-19.3.0-4775.g7f24e589.el9 214 kB/s | 49 kB 00:00 2024-09-14T06:26:53.135 INFO:teuthology.orchestra.run.smithi001.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-14T06:26:53.178 INFO:teuthology.orchestra.run.smithi001.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.3 MB/s | 279 kB 00:00 2024-09-14T06:26:53.228 INFO:teuthology.orchestra.run.smithi001.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-09-14T06:26:53.253 INFO:teuthology.orchestra.run.smithi001.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 120 kB/s | 14 kB 00:00 2024-09-14T06:26:53.278 INFO:teuthology.orchestra.run.smithi001.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-14T06:26:53.303 INFO:teuthology.orchestra.run.smithi001.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-14T06:26:53.329 INFO:teuthology.orchestra.run.smithi001.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-14T06:26:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-09-14T06:26:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-14T06:26:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:26:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:26:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:26:53.556 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:26:53.557 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:26:54.075 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:26:54.252 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T06:26:54.307 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T06:26:54.364 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T06:26:54.398 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T06:26:54.441 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T06:26:54.547 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T06:26:54.659 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T06:26:54.710 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 1/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 2/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 3/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-14T06:26:54.979 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-14T06:26:54.980 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-14T06:26:55.304 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-14T06:26:55.305 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-14T06:26:55.306 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-14T06:26:55.306 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T06:26:55.306 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T06:26:55.306 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:55.306 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:26:55.472 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T06:26:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T06:26:55.555 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T06:26:55.576 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T06:26:55.605 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T06:26:55.633 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T06:26:56.135 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:12 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:26:56.269 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================= 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================= 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 7.4 M 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T06:26:56.270 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================= 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2024-09-14T06:26:56.271 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:26:56.272 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 38 M 2024-09-14T06:26:56.272 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 212 M 2024-09-14T06:26:56.272 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:26:56.428 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 1/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T06:26:56.429 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T06:26:56.602 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): libgfortran-11.5.0-2.el9.x86_64.rpm 5.0 MB/s | 801 kB 00:00 2024-09-14T06:26:56.627 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-14T06:26:56.702 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-09-14T06:26:56.744 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-14T06:26:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T06:26:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:56.769 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:26:56.769 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T06:26:56.770 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T06:26:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T06:26:56.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:56.771 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:26:56.794 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 772 kB/s | 38 kB 00:00 2024-09-14T06:26:56.861 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-09-14T06:26:56.962 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): ceph-mgr-diskprediction-local-19.3.0-47 14 MB/s | 7.4 MB 00:00 2024-09-14T06:26:56.987 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-14T06:26:57.021 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T06:26:57.029 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-14T06:26:57.121 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 15 MB/s | 4.9 MB 00:00 2024-09-14T06:26:57.372 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-09-14T06:26:57.447 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-14T06:26:57.448 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:26:57.448 INFO:teuthology.orchestra.run.smithi170.stdout:Total 32 MB/s | 38 MB 00:01 2024-09-14T06:26:57.615 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:26 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:26:57.755 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:26:57.755 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 154 k 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:26:57.756 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:26:57.757 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-09-14T06:26:57.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:26:57.757 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 7.0 M 2024-09-14T06:26:57.757 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 33 M 2024-09-14T06:26:57.757 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:26:57.819 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:26:57.845 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:26:57.845 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:26:58.293 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:26:58.293 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:26:58.372 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4775.g7f24e589.e 739 kB/s | 154 kB 00:00 2024-09-14T06:26:58.423 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-14T06:26:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-14T06:26:58.473 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-14T06:26:58.990 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 300 kB/s | 248 kB 00:00 2024-09-14T06:26:59.051 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:26:59.115 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T06:26:59.154 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T06:26:59.252 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T06:26:59.297 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T06:26:59.480 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T06:26:59.512 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T06:26:59.597 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T06:27:00.267 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T06:27:00.376 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T06:27:01.598 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T06:27:01.815 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.6 MB/s | 6.0 MB 00:03 2024-09-14T06:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:01.817 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.7 MB/s | 7.0 MB 00:04 2024-09-14T06:27:01.891 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:01.903 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:01.903 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:02.011 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:02.012 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:02.343 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:02.538 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T06:27:02.585 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T06:27:02.638 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T06:27:02.671 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T06:27:03.136 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T06:27:03.231 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T06:27:03.307 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T06:27:03.337 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T06:27:04.029 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T06:27:04.030 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/6 2024-09-14T06:27:04.030 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T06:27:04.030 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T06:27:04.030 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T06:27:04.030 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T06:27:04.319 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T06:27:04.319 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:04.319 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:04.320 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:04.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse 2024-09-14T06:27:05.164 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:34 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:05.281 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T06:27:05.281 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 1/12 2024-09-14T06:27:05.281 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T06:27:05.281 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T06:27:05.281 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T06:27:05.281 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T06:27:05.281 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T06:27:05.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T06:27:05.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T06:27:05.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T06:27:05.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T06:27:05.282 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T06:27:05.307 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:05.307 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:05.307 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 900 k 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-09-14T06:27:05.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:05.309 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 979 k 2024-09-14T06:27:05.309 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.9 M 2024-09-14T06:27:05.309 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:27:05.717 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): ceph-fuse-19.3.0-4775.g7f24e589.el9.x86_ 3.3 MB/s | 900 kB 00:00 2024-09-14T06:27:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T06:27:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:05.771 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:05.771 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:05.772 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 125 kB/s | 80 kB 00:00 2024-09-14T06:27:06.084 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:06.085 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.2 MB/s | 979 kB 00:00 2024-09-14T06:27:06.099 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:06.125 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:06.125 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:06.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2024-09-14T06:27:06.214 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:06.214 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:06.410 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:06.548 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T06:27:06.627 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T06:27:06.746 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:22 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:06.881 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:06.882 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 49 k 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T06:27:06.883 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2024-09-14T06:27:06.884 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:06.885 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.2 M 2024-09-14T06:27:06.885 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2024-09-14T06:27:06.885 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:07.316 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T06:27:07.316 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/2 2024-09-14T06:27:07.413 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 164 kB/s | 19 kB 00:00 2024-09-14T06:27:07.430 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 26 kB 00:00 2024-09-14T06:27:07.447 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): ceph-mgr-rook-19.3.0-4775.g7f24e589.el9 327 kB/s | 49 kB 00:00 2024-09-14T06:27:07.547 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-14T06:27:07.572 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-14T06:27:07.598 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-14T06:27:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T06:27:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:07.603 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T06:27:07.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:07.603 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:07.631 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 551 kB/s | 32 kB 00:00 2024-09-14T06:27:07.657 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-14T06:27:07.674 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 187 kB/s | 14 kB 00:00 2024-09-14T06:27:07.707 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.0 MB/s | 59 kB 00:00 2024-09-14T06:27:07.732 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.2 MB/s | 220 kB 00:00 2024-09-14T06:27:07.758 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-14T06:27:07.800 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-14T06:27:07.803 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:07.803 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-14T06:27:07.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-volume 2024-09-14T06:27:07.867 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:07.883 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:07.883 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:07.974 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:07.974 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:08.459 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:37 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:08.496 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:08.597 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 285 k 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 285 k 2024-09-14T06:27:08.598 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.3 M 2024-09-14T06:27:08.599 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:27:08.673 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T06:27:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T06:27:08.798 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T06:27:08.829 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-volume-19.3.0-4775.g7f24e589.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-14T06:27:08.830 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:08.830 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-14T06:27:08.830 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:08.835 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T06:27:08.839 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:08.840 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:08.873 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:08.874 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:08.876 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T06:27:08.987 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T06:27:09.088 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T06:27:09.107 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:09.145 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T06:27:09.173 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:27:09.190 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:27:09.190 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:27:09.190 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-14T06:27:09.190 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:09.866 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T06:27:09.926 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T06:27:09.960 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T06:27:10.008 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T06:27:10.032 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T06:27:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:27:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:10.258 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:27:10.258 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:10.258 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:10.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel 2024-09-14T06:27:10.870 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 1/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T06:27:10.871 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T06:27:11.170 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:40 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T06:27:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:11.220 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:27:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T06:27:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T06:27:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T06:27:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T06:27:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:11.221 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:11.311 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 126 k 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:11.312 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 126 k 2024-09-14T06:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 456 k 2024-09-14T06:27:11.313 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:27:11.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T06:27:11.535 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-19.3.0-4775.g7f24e589.el9.x86_64 569 kB/s | 126 kB 00:00 2024-09-14T06:27:11.536 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:11.536 INFO:teuthology.orchestra.run.smithi001.stdout:Total 564 kB/s | 126 kB 00:00 2024-09-14T06:27:11.537 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:11.542 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:11.543 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:11.610 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:11.611 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:11.765 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:11.870 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:12.038 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:28 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:12.172 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:12.173 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:12.173 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:27:12.173 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:12.173 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 154 k 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2024-09-14T06:27:12.174 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:12.175 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 7.0 M 2024-09-14T06:27:12.175 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 33 M 2024-09-14T06:27:12.175 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:12.331 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:12.608 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:12.608 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:12.608 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:12.608 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:12.608 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:12.608 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:12.644 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-14T06:27:12.678 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4775.g7f24e589.e 640 kB/s | 154 kB 00:00 2024-09-14T06:27:12.737 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-14T06:27:12.762 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 686 kB/s | 58 kB 00:00 2024-09-14T06:27:12.787 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-14T06:27:12.864 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2 2024-09-14T06:27:13.458 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:13.551 INFO:teuthology.orchestra.run.smithi001.stdout:Package libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:13.598 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:13.599 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:13.600 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:13.674 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs-devel 2024-09-14T06:27:14.265 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:43 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:14.406 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:14.407 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:14.407 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-14T06:27:14.407 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:14.407 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:27:14.407 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 32 k 2024-09-14T06:27:14.407 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:14.407 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:27:14.408 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:14.408 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:27:14.408 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:14.408 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 32 k 2024-09-14T06:27:14.408 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 142 k 2024-09-14T06:27:14.408 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:27:14.564 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs-devel-19.3.0-4775.g7f24e589.el9.x86_6 204 kB/s | 32 kB 00:00 2024-09-14T06:27:14.565 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:14.565 INFO:teuthology.orchestra.run.smithi001.stdout:Total 202 kB/s | 32 kB 00:00 2024-09-14T06:27:14.565 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:14.569 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:14.569 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:14.601 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:14.601 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:14.692 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:14.815 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:14.946 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.4 MB/s | 6.0 MB 00:02 2024-09-14T06:27:14.952 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:14.953 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.5 MB/s | 7.0 MB 00:02 2024-09-14T06:27:15.024 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:15.035 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:15.035 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:15.136 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:15.138 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:15.183 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:15.415 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:15.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:15.416 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:15.416 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:15.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:15.416 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:15.440 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:15.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2 2024-09-14T06:27:15.609 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T06:27:15.656 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T06:27:16.173 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T06:27:16.185 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:16.258 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T06:27:16.279 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:16.324 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:16.325 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:16.325 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:16.326 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T06:27:16.352 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T06:27:16.389 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1 2024-09-14T06:27:16.980 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:17.073 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:17.118 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:17.119 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:17.119 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:17.165 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T06:27:17.165 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/6 2024-09-14T06:27:17.165 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T06:27:17.166 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T06:27:17.166 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T06:27:17.166 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T06:27:17.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados 2024-09-14T06:27:17.449 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T06:27:17.449 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:17.449 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:17.450 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:17.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2024-09-14T06:27:17.781 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:46 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:17.875 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:17.922 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:17.923 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:17.924 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:17.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw 2024-09-14T06:27:18.298 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:34 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:18.432 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:18.433 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 900 k 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:18.434 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:18.435 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2024-09-14T06:27:18.435 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:18.435 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 979 k 2024-09-14T06:27:18.435 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.9 M 2024-09-14T06:27:18.435 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:18.581 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:47 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:18.674 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:18.720 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:18.721 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:18.721 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:18.740 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): ceph-fuse-19.3.0-4775.g7f24e589.el9.x86_ 3.4 MB/s | 900 kB 00:00 2024-09-14T06:27:18.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs 2024-09-14T06:27:18.865 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 208 kB/s | 80 kB 00:00 2024-09-14T06:27:18.865 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:18.866 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.2 MB/s | 979 kB 00:00 2024-09-14T06:27:18.877 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:18.899 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:18.899 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:18.979 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:18.980 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:19.175 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:19.317 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T06:27:19.387 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:48 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:19.388 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T06:27:19.482 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:19.528 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:19.529 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:19.529 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:19.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd 2024-09-14T06:27:20.076 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T06:27:20.077 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/2 2024-09-14T06:27:20.196 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:49 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:20.289 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:20.335 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:20.337 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:20.337 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:20.361 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T06:27:20.361 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:20.361 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:20.362 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T06:27:20.362 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:20.362 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:20.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse 2024-09-14T06:27:20.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-volume 2024-09-14T06:27:21.007 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:50 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:21.148 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 85 k 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:27:21.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:21.150 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 85 k 2024-09-14T06:27:21.150 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 231 k 2024-09-14T06:27:21.150 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:27:21.201 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:37 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:21.341 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 285 k 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:21.342 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:27:21.343 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:21.343 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 285 k 2024-09-14T06:27:21.343 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.3 M 2024-09-14T06:27:21.343 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:21.343 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-19.3.0-4775.g7f24e589.el9.x86_64.rpm 442 kB/s | 85 kB 00:00 2024-09-14T06:27:21.344 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:21.344 INFO:teuthology.orchestra.run.smithi001.stdout:Total 438 kB/s | 85 kB 00:00 2024-09-14T06:27:21.344 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:21.351 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:21.351 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:21.418 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:21.419 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:21.554 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-volume-19.3.0-4775.g7f24e589.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-14T06:27:21.555 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:21.555 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-14T06:27:21.555 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:21.564 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:21.564 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:21.567 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:21.592 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:21.592 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:21.676 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:21.810 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:21.890 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:27:21.906 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:27:21.907 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:27:21.907 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-14T06:27:21.907 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:22.064 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:22.332 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:22.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:22.332 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:22.332 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:22.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:22.332 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:22.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-09-14T06:27:22.915 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T06:27:22.915 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:22.915 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:22.915 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T06:27:22.915 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:22.915 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:23.213 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:52 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:23.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2024-09-14T06:27:23.354 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:27:23.355 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:23.356 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:27:23.356 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:23.356 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 3.2 M 2024-09-14T06:27:23.356 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 14 M 2024-09-14T06:27:23.356 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:27:23.829 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:39 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:23.838 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-19.3.0-4775.g7f24e589.el9.x86_64.rpm 6.6 MB/s | 3.2 MB 00:00 2024-09-14T06:27:23.839 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:23.839 INFO:teuthology.orchestra.run.smithi001.stdout:Total 6.6 MB/s | 3.2 MB 00:00 2024-09-14T06:27:23.839 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:23.849 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:23.849 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:23.936 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:23.936 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:23.963 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:23.964 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-09-14T06:27:23.964 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:23.964 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 126 k 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 126 k 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 456 k 2024-09-14T06:27:23.965 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:24.144 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-19.3.0-4775.g7f24e589.el9.x86_64 710 kB/s | 126 kB 00:00 2024-09-14T06:27:24.144 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:24.144 INFO:teuthology.orchestra.run.smithi170.stdout:Total 703 kB/s | 126 kB 00:00 2024-09-14T06:27:24.144 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:24.150 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:24.150 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:24.214 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:24.214 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:24.367 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:24.475 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:24.482 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:27:24.482 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T06:27:24.482 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T06:27:24.482 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T06:27:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:24.930 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:25.201 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:25.202 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:25.202 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:25.202 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:25.202 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:25.202 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:25.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2024-09-14T06:27:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:25.572 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:25.572 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:25.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd 2024-09-14T06:27:26.062 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:26.158 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:26.205 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:26.206 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:26.206 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:26.282 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2024-09-14T06:27:26.419 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:55 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:26.560 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 171 k 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-14T06:27:26.561 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-14T06:27:26.562 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:26.562 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 171 k 2024-09-14T06:27:26.562 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 490 k 2024-09-14T06:27:26.562 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-14T06:27:26.767 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-19.3.0-4775.g7f24e589.el9.x86_64.rpm 835 kB/s | 171 kB 00:00 2024-09-14T06:27:26.767 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:26.767 INFO:teuthology.orchestra.run.smithi001.stdout:Total 828 kB/s | 171 kB 00:00 2024-09-14T06:27:26.768 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-14T06:27:26.776 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-14T06:27:26.776 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-14T06:27:26.843 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-14T06:27:26.844 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-14T06:27:26.879 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:27.018 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-14T06:27:27.019 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 32 k 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:27.020 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:27:27.021 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:27.021 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 32 k 2024-09-14T06:27:27.021 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 142 k 2024-09-14T06:27:27.021 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:27.117 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:27.190 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-19.3.0-4775.g7f24e589.el9.x86_6 187 kB/s | 32 kB 00:00 2024-09-14T06:27:27.191 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:27.191 INFO:teuthology.orchestra.run.smithi170.stdout:Total 186 kB/s | 32 kB 00:00 2024-09-14T06:27:27.192 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:27.196 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:27.196 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:27.215 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:27.215 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:27.297 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:27.397 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:27.555 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:27.784 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:27.819 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:27.819 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:27.819 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-14T06:27:27.819 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:27.819 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:27.819 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:28.038 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:28.039 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:28.039 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:28.039 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:28.039 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:28.039 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:28.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-09-14T06:27:28.223 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2024-09-14T06:27:28.670 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:57 ago on Sat 14 Sep 2024 06:25:31 AM UTC. 2024-09-14T06:27:28.764 INFO:teuthology.orchestra.run.smithi001.stdout:Package rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:28.786 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:44 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:28.811 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-14T06:27:28.812 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-14T06:27:28.812 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-14T06:27:28.876 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:28.882 DEBUG:teuthology.parallel:result is None 2024-09-14T06:27:28.919 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:28.920 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:28.920 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:28.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2024-09-14T06:27:29.553 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:29.643 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:29.687 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:29.688 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:29.688 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:29.760 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2024-09-14T06:27:30.322 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:46 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:30.414 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:30.458 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:30.459 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:30.459 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:30.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2024-09-14T06:27:31.095 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:47 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:31.184 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:31.227 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:31.228 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:31.228 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:31.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2024-09-14T06:27:31.873 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:47 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:31.965 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:32.009 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:32.010 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:32.010 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:32.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2024-09-14T06:27:32.637 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:48 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:32.728 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:32.772 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:32.772 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:32.773 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:32.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2024-09-14T06:27:33.405 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:49 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:33.538 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 85 k 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 85 k 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 231 k 2024-09-14T06:27:33.539 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:33.761 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-19.3.0-4775.g7f24e589.el9.x86_64.rpm 384 kB/s | 85 kB 00:00 2024-09-14T06:27:33.762 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:33.762 INFO:teuthology.orchestra.run.smithi170.stdout:Total 382 kB/s | 85 kB 00:00 2024-09-14T06:27:33.762 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:33.769 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:33.769 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:33.833 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:33.833 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:33.984 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:34.093 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:34.520 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:34.800 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:34.800 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:34.800 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:34.801 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:34.801 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:34.801 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:35.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2024-09-14T06:27:35.668 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:51 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:35.805 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:35.806 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:27:35.807 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.2 M 2024-09-14T06:27:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 14 M 2024-09-14T06:27:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:36.139 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-19.3.0-4775.g7f24e589.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-14T06:27:36.140 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:36.141 INFO:teuthology.orchestra.run.smithi170.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-14T06:27:36.141 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:36.150 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:36.150 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:36.231 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:36.231 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:36.675 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:36.749 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:36.774 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:36.774 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T06:27:36.774 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T06:27:36.774 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T06:27:36.774 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T06:27:36.775 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:37.981 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:37.981 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:37.981 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:37.981 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:37.981 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:37.981 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:38.252 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2024-09-14T06:27:38.840 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:54 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:38.974 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:38.974 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 171 k 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 171 k 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 490 k 2024-09-14T06:27:38.975 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-09-14T06:27:39.194 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-19.3.0-4775.g7f24e589.el9.x86_64.rpm 780 kB/s | 171 kB 00:00 2024-09-14T06:27:39.195 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-09-14T06:27:39.195 INFO:teuthology.orchestra.run.smithi170.stdout:Total 775 kB/s | 171 kB 00:00 2024-09-14T06:27:39.196 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-09-14T06:27:39.203 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-09-14T06:27:39.204 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-09-14T06:27:39.267 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-09-14T06:27:39.267 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-09-14T06:27:39.427 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-09-14T06:27:39.524 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:39.966 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T06:27:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:40.219 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-09-14T06:27:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T06:27:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:40.219 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:40.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2024-09-14T06:27:41.047 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:57 ago on Sat 14 Sep 2024 06:25:44 AM UTC. 2024-09-14T06:27:41.140 INFO:teuthology.orchestra.run.smithi170.stdout:Package rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T06:27:41.187 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-09-14T06:27:41.188 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-09-14T06:27:41.188 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-09-14T06:27:41.260 DEBUG:teuthology.parallel:result is None 2024-09-14T06:27:41.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T06:27:41.410 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T06:27:41.437 INFO:teuthology.orchestra.run.smithi001.stdout:19.3.0-4775.g7f24e589.el9 2024-09-14T06:27:41.438 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4775.g7f24e589.el9 2024-09-14T06:27:41.438 INFO:teuthology.task.install:The correct ceph version 19.3.0-4775.g7f24e589 is installed. 2024-09-14T06:27:41.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T06:27:41.584 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T06:27:41.605 INFO:teuthology.orchestra.run.smithi170.stdout:19.3.0-4775.g7f24e589.el9 2024-09-14T06:27:41.606 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4775.g7f24e589.el9 2024-09-14T06:27:41.606 INFO:teuthology.task.install:The correct ceph version 19.3.0-4775.g7f24e589 is installed. 2024-09-14T06:27:41.608 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-14T06:27:41.608 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:41.608 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T06:27:41.639 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:41.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T06:27:41.672 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-14T06:27:41.672 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:41.672 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T06:27:41.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T06:27:41.778 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:41.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T06:27:41.803 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T06:27:41.868 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-14T06:27:41.868 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:41.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T06:27:41.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T06:27:41.964 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:41.964 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T06:27:41.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T06:27:42.053 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-14T06:27:42.054 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:42.054 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T06:27:42.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T06:27:42.145 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:42.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T06:27:42.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T06:27:42.234 INFO:teuthology.run_tasks:Running task ceph... 2024-09-14T06:27:42.311 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-14T06:27:42.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-09-14T06:27:42.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2024-09-14T06:27:42.338 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-14T06:27:42.338 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T06:27:42.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T06:27:42.405 INFO:tasks.ceph:Creating extra log directories... 2024-09-14T06:27:42.405 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T06:27:42.447 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T06:27:42.484 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-14T06:27:42.484 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-14T06:27:42.484 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380', 'branch': 'main', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900380', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 950, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5839, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'sleep_before_teardown': 0, 'subset': '99/128', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/1YN9cdEByEgxDm3lqN2v8GTzP/mHK4T/7oMWEBifSOh9bmZZXO3sHGPVW1dKEYnMLNg2KODvzIA8Wq6ZWRz0=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+vaI4OBZX+ycCfdIr6NI27KSkRiUV6FBKOMObmKr/VHKf+FKu+UhLydleKQ0YcZ1uFNsOnPH7i/3IMTt+oxTw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_20:16:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-14T06:27:42.485 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-14T06:27:42.513 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-14T06:27:42.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T06:27:42.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T06:27:42.594 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:42.594 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T06:27:42.649 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T06:27:42.649 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-09-14T06:27:42.704 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-14 06:27:24.742192064 +0000 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-14 06:24:51.199879956 +0000 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-14 06:24:51.199879956 +0000 2024-09-14T06:27:42.705 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-14 06:24:51.199879956 +0000 2024-09-14T06:27:42.705 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T06:27:42.769 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-14T06:27:42.769 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-14T06:27:42.769 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.3716e-05 s, 5.5 MB/s 2024-09-14T06:27:42.773 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T06:27:42.830 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-09-14T06:27:42.886 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-14 06:27:24.742192064 +0000 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-14 06:24:51.525875506 +0000 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-14 06:24:51.525875506 +0000 2024-09-14T06:27:42.887 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-14 06:24:51.525875506 +0000 2024-09-14T06:27:42.887 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T06:27:42.951 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-14T06:27:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-14T06:27:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000100279 s, 5.1 MB/s 2024-09-14T06:27:42.953 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T06:27:43.009 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-09-14T06:27:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T06:27:43.064 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:27:43.065 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-14T06:27:43.065 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:27:43.065 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:27:43.065 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-14 06:27:24.743192050 +0000 2024-09-14T06:27:43.065 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-14 06:24:51.845871138 +0000 2024-09-14T06:27:43.065 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-14 06:24:51.845871138 +0000 2024-09-14T06:27:43.065 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-14 06:24:51.845871138 +0000 2024-09-14T06:27:43.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T06:27:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-14T06:27:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-14T06:27:43.129 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000103204 s, 5.0 MB/s 2024-09-14T06:27:43.131 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T06:27:43.187 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-14 06:27:24.743192050 +0000 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-14 06:24:52.172866675 +0000 2024-09-14T06:27:43.243 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-14 06:24:52.172866675 +0000 2024-09-14T06:27:43.244 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-14 06:24:52.172866675 +0000 2024-09-14T06:27:43.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T06:27:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-14T06:27:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-14T06:27:43.307 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 7.9559e-05 s, 6.4 MB/s 2024-09-14T06:27:43.308 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T06:27:43.365 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:27:43.365 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T06:27:43.365 INFO:tasks.ceph:Generating config... 2024-09-14T06:27:43.366 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-14T06:27:43.366 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-14T06:27:43.366 INFO:tasks.ceph:[client] log to stderr = False 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.367 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.368 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.368 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.368 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.368 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.368 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-14T06:27:43.368 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-14T06:27:43.368 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-14T06:27:43.368 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-14T06:27:43.368 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-14T06:27:43.368 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-14T06:27:43.368 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-14T06:27:43.368 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-14T06:27:43.368 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-14T06:27:43.369 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-14T06:27:43.370 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-14T06:27:43.370 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-14T06:27:43.370 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-14T06:27:43.370 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-14T06:27:43.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-14T06:27:43.454 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/cluster1.keyring 2024-09-14T06:27:43.457 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-14T06:27:43.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T06:27:43.567 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.1')] 2024-09-14T06:27:43.567 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-14T06:27:43.568 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:43.568 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T06:27:43.623 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:43.704 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:43.704 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid 5c072185-95f2-48ae-9e2f-90187671cc6f 2024-09-14T06:27:43.704 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 5c072185-95f2-48ae-9e2f-90187671cc6f 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-09-14T06:27:43.704354+0000 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-09-14T06:27:43.704354+0000 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-09-14T06:27:43.705 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-14T06:27:43.706 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T06:27:43.761 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 5c072185-95f2-48ae-9e2f-90187671cc6f... 2024-09-14T06:27:43.762 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-14T06:27:43.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-14T06:27:43.846 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-09-14T06:27:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-09-14T06:27:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T06:27:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-09-14T06:27:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-09-14T06:27:43.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T06:27:43.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T06:27:43.848 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = 172.21.15.1 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-09-14T06:27:43.849 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on pool no app = False 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 5000 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 5c072185-95f2-48ae-9e2f-90187671cc6f 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-09-14T06:27:43.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.851 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1/20 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1/20 2024-09-14T06:27:43.852 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 4/10 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: osd memory target = 1610612736 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-14T06:27:43.853 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-09-14T06:27:43.854 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd = 20 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd_mirror = 15 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: log to stderr = False 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: rbd default features = 125 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.0] 2024-09-14T06:27:43.856 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.1] 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.2] 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.3] 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.4] 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.857 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.858 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.5] 2024-09-14T06:27:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.858 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.6] 2024-09-14T06:27:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.858 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.859 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2024-09-14T06:27:43.860 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.861 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = 172.21.15.1 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on pool no app = False 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 5000 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = 5c072185-95f2-48ae-9e2f-90187671cc6f 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2024-09-14T06:27:43.862 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2024-09-14T06:27:43.863 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2024-09-14T06:27:43.864 INFO:teuthology.orchestra.run.smithi170.stdout: bdev async discard = True 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: bdev enable discard = True 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 1/20 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 1/20 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 4/10 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2024-09-14T06:27:43.865 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: osd memory target = 1610612736 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.866 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T06:27:43.867 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2024-09-14T06:27:43.868 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd = 20 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd_mirror = 15 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: log to stderr = False 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: rbd default features = 125 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2024-09-14T06:27:43.869 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.4] 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.5] 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.6] 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:27:43.870 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:27:43.872 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-14T06:27:43.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-14T06:27:43.968 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-14T06:27:43.968 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:43.968 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-14T06:27:43.984 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:43.984 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-14T06:27:44.039 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T06:27:44.040 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:44.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-14T06:27:44.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T06:27:44.116 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:44.116 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:44.172 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:27:44.172 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:44.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-14T06:27:44.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T06:27:44.213 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:44.213 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:44.272 INFO:tasks.ceph:Setting up mon nodes... 2024-09-14T06:27:44.272 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-14T06:27:44.272 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-14T06:27:44.325 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-14T06:27:44.328 INFO:tasks.ceph:Setting up mds nodes... 2024-09-14T06:27:44.328 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-14T06:27:44.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-14T06:27:44.371 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-14T06:27:44.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-14T06:27:44.439 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-14T06:27:44.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-14T06:27:44.496 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-14T06:27:44.509 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-14T06:27:44.591 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-14T06:27:44.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-14T06:27:44.685 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-14T06:27:44.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-14T06:27:44.778 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-14T06:27:44.790 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-14T06:27:44.872 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-14T06:27:44.885 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-14T06:27:44.965 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-14T06:27:44.978 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-14T06:27:45.059 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-14T06:27:45.071 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-14T06:27:45.072 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T06:27:45.072 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-14T06:27:45.101 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:27:45.101 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-14T06:27:45.101 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T06:27:45.101 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T06:27:45.171 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T06:27:45.203 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-14T06:27:45.205 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-14T06:27:45.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-14T06:27:45.278 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-14T06:27:45.350 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-14T06:27:45.418 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:27:45.418 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-14T06:27:45.418 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T06:27:45.418 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T06:27:45.483 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T06:27:45.484 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T06:27:45.516 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-14T06:27:45.518 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-14T06:27:45.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-14T06:27:45.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-14T06:27:45.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-14T06:27:45.725 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:27:45.725 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-14T06:27:45.725 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T06:27:45.725 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T06:27:45.790 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T06:27:45.823 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-14T06:27:45.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-14T06:27:45.825 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-14T06:27:45.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-14T06:27:45.965 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:46.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:46.056+0000 7f8d9fc28640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-14T06:27:46.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:46.056+0000 7f8d9fc28640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-14T06:27:46.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:46.056+0000 7f8d9fc28640 -1 bdev(0x560394505800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-14T06:27:46.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:46.056+0000 7f8d9fc28640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-14T06:27:49.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-14T06:27:49.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:49.351+0000 7f1b75ee8640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-14T06:27:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:49.351+0000 7f1b75ee8640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-14T06:27:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:49.351+0000 7f1b75ee8640 -1 bdev(0x556c6aa83800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-14T06:27:49.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:49.351+0000 7f1b75ee8640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-14T06:27:52.269 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-14T06:27:52.300 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:52.391+0000 7f1971530640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-14T06:27:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:52.391+0000 7f1971530640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-14T06:27:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:52.391+0000 7f1971530640 -1 bdev(0x5607a2707800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-14T06:27:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-14T06:27:52.391+0000 7f1971530640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-14T06:27:55.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-14T06:27:55.568 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-14T06:27:55.569 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:55.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-14T06:27:55.636 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:55.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-14T06:27:55.702 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:55.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-14T06:27:55.768 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:55.768 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-14T06:27:55.834 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:55.834 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-14T06:27:55.891 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:55.891 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-14T06:27:55.910 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:55.910 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-14T06:27:55.968 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:55.968 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-14T06:27:56.026 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:56.026 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-14T06:27:56.084 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:56.084 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-14T06:27:56.141 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:56.141 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-14T06:27:56.198 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:56.198 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-14T06:27:56.255 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:27:56.255 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-14T06:27:56.313 INFO:tasks.ceph:Adding keys to all mons... 2024-09-14T06:27:56.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-14T06:27:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-09-14T06:27:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVmdaBjExAAeXiiHWWNgKbfrXT/P2d08w== 2024-09-14T06:27:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-09-14T06:27:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDiLOVmszZ9AxAAjt+JDKC9nclWjoSkODarzg== 2024-09-14T06:27:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-09-14T06:27:56.340 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDlLOVmGOj/FBAAmfdPESzmxZXttY7lyAf6Gw== 2024-09-14T06:27:56.340 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDoLOVmeMthFxAAvu+exYvTFB21A+rLqWV1OQ== 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVmctIaFhAA5o2FdgBLXxekmXYhbER0pQ== 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror] 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVmvIAsGhAA870jo170wegxMn7oXaukig== 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.0] 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVm57OXHRAAQU+bf8Y88H87JCgpsVlGFw== 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.1] 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVmeMdEIxAAPMZ/fvIM2co2eOJMbZcdgQ== 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.2] 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVmFTjRKBAAu/+Nn3rD6BwPc6kEFkYVhw== 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.3] 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVmETtfLhAAIogfIZo650stp+MZ6429qQ== 2024-09-14T06:27:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.4] 2024-09-14T06:27:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVm6yv/MxAAtji6d0LTAhJd4j570uFv9A== 2024-09-14T06:27:56.342 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.5] 2024-09-14T06:27:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDgLOVmN3+PORAAWpdNTj3dV5RoEZG7J9/Okw== 2024-09-14T06:27:56.342 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.6] 2024-09-14T06:27:56.342 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDhLOVmgCmEAxAAT89M6/A1O61xocQ5XQb94A== 2024-09-14T06:27:56.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T06:27:56.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T06:27:56.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T06:27:56.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T06:27:56.602 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:56.657 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:56.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:56.768 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:56.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:56.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:56.935 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:56.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:57.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:27:57.102 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-14T06:27:57.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-14T06:27:57.128 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-14T06:27:57.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-14T06:27:57.251 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T06:27:57.307 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-14T06:27:57.307 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-14T06:27:57.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-14T06:27:57.349 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-14T06:27:57.349 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-14T06:27:57.350 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-14T06:27:57.350 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-14T06:27:57.352 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-14T06:27:57.352 DEBUG:tasks.ceph:set 0 configs 2024-09-14T06:27:57.352 INFO:tasks.ceph:Setting crush tunables to default 2024-09-14T06:27:57.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-14T06:27:57.530 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-09-14T06:27:57.541 INFO:tasks.ceph:check_enable_crimson: False 2024-09-14T06:27:57.541 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-14T06:27:57.541 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:57.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-14T06:27:57.567 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:57.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-14T06:27:57.633 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:27:57.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-14T06:27:57.700 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd new 71e380e7-23ee-4d9f-a5dc-914c6a7fb873 0 2024-09-14T06:27:57.924 INFO:teuthology.orchestra.run.smithi001.stdout:0 2024-09-14T06:27:57.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd new d8033edb-e276-4d40-a9a4-32ec43ac783b 1 2024-09-14T06:27:58.117 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-09-14T06:27:58.128 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd new 709cab32-6621-4a1e-8a40-e2a732d2b50c 2 2024-09-14T06:27:58.310 INFO:teuthology.orchestra.run.smithi001.stdout:2 2024-09-14T06:27:58.321 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-14T06:27:58.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-14T06:27:58.324 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-14T06:27:58.324 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-14T06:27:58.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-14T06:27:58.326 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-14T06:27:58.326 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-14T06:27:58.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-14T06:27:58.328 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-14T06:27:58.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T06:27:58.506 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:27:58.506 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"fsid":"5c072185-95f2-48ae-9e2f-90187671cc6f","created":"2024-09-14T06:27:57.438728+0000","modified":"2024-09-14T06:27:58.309768+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T06:27:58.309768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"71e380e7-23ee-4d9f-a5dc-914c6a7fb873","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8033edb-e276-4d40-a9a4-32ec43ac783b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"709cab32-6621-4a1e-8a40-e2a732d2b50c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:27:58.517 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-14T06:27:58.517 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-14T06:27:58.820 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T06:27:59.117 INFO:teuthology.misc.health.smithi001.stderr:2024-09-14T06:27:59.115+0000 7f88937fe640 0 --2- 172.21.15.1:0/785585740 >> v2:172.21.15.1:3300/0 conn(0x7f8894074500 0x7f88940b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-14T06:27:59.185 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-14T06:27:59.185 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":5,"fsid":"5c072185-95f2-48ae-9e2f-90187671cc6f","created":"2024-09-14T06:27:57.438728+0000","modified":"2024-09-14T06:27:58.309768+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T06:27:58.309768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"71e380e7-23ee-4d9f-a5dc-914c6a7fb873","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8033edb-e276-4d40-a9a4-32ec43ac783b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"709cab32-6621-4a1e-8a40-e2a732d2b50c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:27:59.195 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T06:27:59.284 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:27:59.282+0000 7f643e7b0640 -1 Falling back to public interface 2024-09-14T06:27:59.304 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:27:59.303+0000 7f14f92b0640 -1 Falling back to public interface 2024-09-14T06:27:59.538 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:27:59.536+0000 7ff315a9c640 -1 Falling back to public interface 2024-09-14T06:28:03.026 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-14T06:28:03.025+0000 7f1bbb68f640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-14T06:28:03.111 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:28:03.110+0000 7f14f92b0640 -1 osd.0 0 log_to_monitors true 2024-09-14T06:28:03.365 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:28:03.364+0000 7ff315a9c640 -1 osd.2 0 log_to_monitors true 2024-09-14T06:28:03.388 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:28:03.387+0000 7f643e7b0640 -1 osd.1 0 log_to_monitors true 2024-09-14T06:28:05.498 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T06:28:06.021 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-14T06:28:06.021 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":7,"fsid":"5c072185-95f2-48ae-9e2f-90187671cc6f","created":"2024-09-14T06:27:57.438728+0000","modified":"2024-09-14T06:28:05.026802+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T06:27:58.309768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"71e380e7-23ee-4d9f-a5dc-914c6a7fb873","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8033edb-e276-4d40-a9a4-32ec43ac783b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"709cab32-6621-4a1e-8a40-e2a732d2b50c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:06.040 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T06:28:06.391 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:28:06.389+0000 7f643af53640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T06:28:06.391 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:28:06.390+0000 7f14f523f640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T06:28:06.406 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:28:06.405+0000 7f643255e640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T06:28:06.407 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:28:06.406+0000 7f14ed05e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T06:28:06.429 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:28:06.428+0000 7ff31223f640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T06:28:06.446 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:28:06.445+0000 7ff30984a640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T06:28:12.343 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T06:28:12.646 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-14T06:28:12.647 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"5c072185-95f2-48ae-9e2f-90187671cc6f","created":"2024-09-14T06:27:57.438728+0000","modified":"2024-09-14T06:28:11.046221+0000","last_up_change":"2024-09-14T06:28:07.395120+0000","last_in_change":"2024-09-14T06:27:58.309768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:08.189460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"71e380e7-23ee-4d9f-a5dc-914c6a7fb873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6809","nonce":1600313121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6811","nonce":1600313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6815","nonce":1600313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6813","nonce":1600313121}]},"public_addr":"172.21.15.1:6809/1600313121","cluster_addr":"172.21.15.1:6811/1600313121","heartbeat_back_addr":"172.21.15.1:6815/1600313121","heartbeat_front_addr":"172.21.15.1:6813/1600313121","state":["exists","up"]},{"osd":1,"uuid":"d8033edb-e276-4d40-a9a4-32ec43ac783b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6801","nonce":1512690988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6803","nonce":1512690988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6807","nonce":1512690988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6805","nonce":1512690988}]},"public_addr":"172.21.15.1:6801/1512690988","cluster_addr":"172.21.15.1:6803/1512690988","heartbeat_back_addr":"172.21.15.1:6807/1512690988","heartbeat_front_addr":"172.21.15.1:6805/1512690988","state":["exists","up"]},{"osd":2,"uuid":"709cab32-6621-4a1e-8a40-e2a732d2b50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6817","nonce":2637205829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6819","nonce":2637205829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6823","nonce":2637205829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6821","nonce":2637205829}]},"public_addr":"172.21.15.1:6817/2637205829","cluster_addr":"172.21.15.1:6819/2637205829","heartbeat_back_addr":"172.21.15.1:6823/2637205829","heartbeat_front_addr":"172.21.15.1:6821/2637205829","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.146261+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.369465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.402451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:12.658 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-14T06:28:12.658 INFO:tasks.ceph:Creating RBD pool 2024-09-14T06:28:12.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-14T06:28:13.055 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'rbd' created 2024-09-14T06:28:13.071 DEBUG:teuthology.orchestra.run.smithi001:> rbd --cluster cluster1 pool init rbd 2024-09-14T06:28:16.077 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-14T06:28:16.078 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 config log 1 --format=json 2024-09-14T06:28:16.079 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-14T06:28:16.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:16.406 INFO:teuthology.orchestra.run.smithi001.stdout:[{"version":4,"timestamp":"2024-09-14T06:28:06.429630+0000","name":"","changes":[{"name":"osd.2/osd_mclock_max_capacity_iops_ssd","new_value":"4133.070196"}]}] 2024-09-14T06:28:16.407 INFO:tasks.ceph_manager:config epoch is 4 2024-09-14T06:28:16.407 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-14T06:28:16.407 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-14T06:28:16.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-14T06:28:16.753 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:16.764 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3307511903},{"type":"v1","addr":"172.21.15.1:6825","nonce":3307511903}]},"active_addr":"172.21.15.1:6825/3307511903","active_change":"2024-09-14T06:28:01.968577+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3816325172}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":376919771}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2917680123}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2235197639}]}]} 2024-09-14T06:28:16.767 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-14T06:28:16.767 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-14T06:28:16.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T06:28:17.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:17.062 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"fsid":"5c072185-95f2-48ae-9e2f-90187671cc6f","created":"2024-09-14T06:27:57.438728+0000","modified":"2024-09-14T06:28:16.061748+0000","last_up_change":"2024-09-14T06:28:07.395120+0000","last_in_change":"2024-09-14T06:27:58.309768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:08.189460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T06:28:12.957687+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"71e380e7-23ee-4d9f-a5dc-914c6a7fb873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6809","nonce":1600313121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6811","nonce":1600313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6815","nonce":1600313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6813","nonce":1600313121}]},"public_addr":"172.21.15.1:6809/1600313121","cluster_addr":"172.21.15.1:6811/1600313121","heartbeat_back_addr":"172.21.15.1:6815/1600313121","heartbeat_front_addr":"172.21.15.1:6813/1600313121","state":["exists","up"]},{"osd":1,"uuid":"d8033edb-e276-4d40-a9a4-32ec43ac783b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6801","nonce":1512690988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6803","nonce":1512690988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6807","nonce":1512690988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6805","nonce":1512690988}]},"public_addr":"172.21.15.1:6801/1512690988","cluster_addr":"172.21.15.1:6803/1512690988","heartbeat_back_addr":"172.21.15.1:6807/1512690988","heartbeat_front_addr":"172.21.15.1:6805/1512690988","state":["exists","up"]},{"osd":2,"uuid":"709cab32-6621-4a1e-8a40-e2a732d2b50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6817","nonce":2637205829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6819","nonce":2637205829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6823","nonce":2637205829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6821","nonce":2637205829}]},"public_addr":"172.21.15.1:6817/2637205829","cluster_addr":"172.21.15.1:6819/2637205829","heartbeat_back_addr":"172.21.15.1:6823/2637205829","heartbeat_front_addr":"172.21.15.1:6821/2637205829","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.146261+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.369465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.402451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:17.075 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-14T06:28:17.075 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T06:28:17.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:17.373 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"fsid":"5c072185-95f2-48ae-9e2f-90187671cc6f","created":"2024-09-14T06:27:57.438728+0000","modified":"2024-09-14T06:28:16.061748+0000","last_up_change":"2024-09-14T06:28:07.395120+0000","last_in_change":"2024-09-14T06:27:58.309768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:08.189460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T06:28:12.957687+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"71e380e7-23ee-4d9f-a5dc-914c6a7fb873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6809","nonce":1600313121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6811","nonce":1600313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6815","nonce":1600313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6813","nonce":1600313121}]},"public_addr":"172.21.15.1:6809/1600313121","cluster_addr":"172.21.15.1:6811/1600313121","heartbeat_back_addr":"172.21.15.1:6815/1600313121","heartbeat_front_addr":"172.21.15.1:6813/1600313121","state":["exists","up"]},{"osd":1,"uuid":"d8033edb-e276-4d40-a9a4-32ec43ac783b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6801","nonce":1512690988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6803","nonce":1512690988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6807","nonce":1512690988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6805","nonce":1512690988}]},"public_addr":"172.21.15.1:6801/1512690988","cluster_addr":"172.21.15.1:6803/1512690988","heartbeat_back_addr":"172.21.15.1:6807/1512690988","heartbeat_front_addr":"172.21.15.1:6805/1512690988","state":["exists","up"]},{"osd":2,"uuid":"709cab32-6621-4a1e-8a40-e2a732d2b50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6817","nonce":2637205829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6819","nonce":2637205829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6823","nonce":2637205829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6821","nonce":2637205829}]},"public_addr":"172.21.15.1:6817/2637205829","cluster_addr":"172.21.15.1:6819/2637205829","heartbeat_back_addr":"172.21.15.1:6823/2637205829","heartbeat_front_addr":"172.21.15.1:6821/2637205829","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.146261+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.369465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.402451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:17.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-14T06:28:17.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-14T06:28:17.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-14T06:28:17.528 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-14T06:28:17.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-14T06:28:17.541 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-14T06:28:17.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-14T06:28:17.542 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-14T06:28:17.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-14T06:28:17.861 INFO:teuthology.orchestra.run.smithi001.stdout:34359738371 2024-09-14T06:28:17.874 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-09-14T06:28:17.889 INFO:teuthology.orchestra.run.smithi001.stdout:34359738371 2024-09-14T06:28:17.902 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2024-09-14T06:28:17.912 INFO:teuthology.orchestra.run.smithi001.stdout:34359738371 2024-09-14T06:28:17.924 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2024-09-14T06:28:18.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-14T06:28:18.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-14T06:28:18.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-14T06:28:19.208 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-14T06:28:19.221 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2024-09-14T06:28:19.221 DEBUG:teuthology.parallel:result is None 2024-09-14T06:28:19.249 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-14T06:28:19.262 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2024-09-14T06:28:19.262 DEBUG:teuthology.parallel:result is None 2024-09-14T06:28:19.294 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-14T06:28:19.306 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2024-09-14T06:28:19.306 DEBUG:teuthology.parallel:result is None 2024-09-14T06:28:19.307 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-14T06:28:19.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T06:28:19.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-14T06:28:19.653 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-14T06:28:17.975000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95128,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020392,"statfs":{"total":289910292480,"available":289812881408,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.919456"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.071845+0000","last_change":"2024-09-14T06:28:16.071845+0000","last_active":"2024-09-14T06:28:16.071845+0000","last_peered":"2024-09-14T06:28:16.071845+0000","last_clean":"2024-09-14T06:28:16.071845+0000","last_became_active":"2024-09-14T06:28:14.066720+0000","last_became_peered":"2024-09-14T06:28:14.066720+0000","last_unstale":"2024-09-14T06:28:16.071845+0000","last_undegraded":"2024-09-14T06:28:16.071845+0000","last_fullsized":"2024-09-14T06:28:16.071845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:18.865038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078953200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.072384+0000","last_change":"2024-09-14T06:28:16.072384+0000","last_active":"2024-09-14T06:28:16.072384+0000","last_peered":"2024-09-14T06:28:16.072384+0000","last_clean":"2024-09-14T06:28:16.072384+0000","last_became_active":"2024-09-14T06:28:14.067568+0000","last_became_peered":"2024-09-14T06:28:14.067568+0000","last_unstale":"2024-09-14T06:28:16.072384+0000","last_undegraded":"2024-09-14T06:28:16.072384+0000","last_fullsized":"2024-09-14T06:28:16.072384+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:25:25.337512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.072011+0000","last_change":"2024-09-14T06:28:16.072011+0000","last_active":"2024-09-14T06:28:16.072011+0000","last_peered":"2024-09-14T06:28:16.072011+0000","last_clean":"2024-09-14T06:28:16.072011+0000","last_became_active":"2024-09-14T06:28:14.065921+0000","last_became_peered":"2024-09-14T06:28:14.065921+0000","last_unstale":"2024-09-14T06:28:16.072011+0000","last_undegraded":"2024-09-14T06:28:16.072011+0000","last_fullsized":"2024-09-14T06:28:16.072011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:45:01.691891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092469000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.072360+0000","last_change":"2024-09-14T06:28:16.072360+0000","last_active":"2024-09-14T06:28:16.072360+0000","last_peered":"2024-09-14T06:28:16.072360+0000","last_clean":"2024-09-14T06:28:16.072360+0000","last_became_active":"2024-09-14T06:28:14.067522+0000","last_became_peered":"2024-09-14T06:28:14.067522+0000","last_unstale":"2024-09-14T06:28:16.072360+0000","last_undegraded":"2024-09-14T06:28:16.072360+0000","last_fullsized":"2024-09-14T06:28:16.072360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:56.670133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056764,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.070458+0000","last_change":"2024-09-14T06:28:16.069659+0000","last_active":"2024-09-14T06:28:16.070458+0000","last_peered":"2024-09-14T06:28:16.070458+0000","last_clean":"2024-09-14T06:28:16.070458+0000","last_became_active":"2024-09-14T06:28:14.065736+0000","last_became_peered":"2024-09-14T06:28:14.065736+0000","last_unstale":"2024-09-14T06:28:16.070458+0000","last_undegraded":"2024-09-14T06:28:16.070458+0000","last_fullsized":"2024-09-14T06:28:16.070458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:29.974020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010548370000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.066424+0000","last_change":"2024-09-14T06:28:16.066424+0000","last_active":"2024-09-14T06:28:16.066424+0000","last_peered":"2024-09-14T06:28:16.066424+0000","last_clean":"2024-09-14T06:28:16.066424+0000","last_became_active":"2024-09-14T06:28:14.065677+0000","last_became_peered":"2024-09-14T06:28:14.065677+0000","last_unstale":"2024-09-14T06:28:16.066424+0000","last_undegraded":"2024-09-14T06:28:16.066424+0000","last_fullsized":"2024-09-14T06:28:16.066424+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:36.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047781599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.066327+0000","last_change":"2024-09-14T06:28:16.066327+0000","last_active":"2024-09-14T06:28:16.066327+0000","last_peered":"2024-09-14T06:28:16.066327+0000","last_clean":"2024-09-14T06:28:16.066327+0000","last_became_active":"2024-09-14T06:28:14.067147+0000","last_became_peered":"2024-09-14T06:28:14.067147+0000","last_unstale":"2024-09-14T06:28:16.066327+0000","last_undegraded":"2024-09-14T06:28:16.066327+0000","last_fullsized":"2024-09-14T06:28:16.066327+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:19:13.309078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042511200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.071663+0000","last_change":"2024-09-14T06:28:16.071663+0000","last_active":"2024-09-14T06:28:16.071663+0000","last_peered":"2024-09-14T06:28:16.071663+0000","last_clean":"2024-09-14T06:28:16.071663+0000","last_became_active":"2024-09-14T06:28:14.065753+0000","last_became_peered":"2024-09-14T06:28:14.065753+0000","last_unstale":"2024-09-14T06:28:16.071663+0000","last_undegraded":"2024-09-14T06:28:16.071663+0000","last_fullsized":"2024-09-14T06:28:16.071663+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:38:56.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066619800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.071108+0000","last_change":"2024-09-14T06:28:10.140562+0000","last_active":"2024-09-14T06:28:16.071108+0000","last_peered":"2024-09-14T06:28:16.071108+0000","last_clean":"2024-09-14T06:28:16.071108+0000","last_became_active":"2024-09-14T06:28:10.139552+0000","last_became_peered":"2024-09-14T06:28:10.139552+0000","last_unstale":"2024-09-14T06:28:16.071108+0000","last_undegraded":"2024-09-14T06:28:16.071108+0000","last_fullsized":"2024-09-14T06:28:16.071108+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_clean_scrub_stamp":"2024-09-14T06:28:09.039132+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:00:29.799410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:28:19.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T06:28:19.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:19.947 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-14T06:28:19.959 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-14T06:28:17.975000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95128,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020392,"statfs":{"total":289910292480,"available":289812881408,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.919456"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.071845+0000","last_change":"2024-09-14T06:28:16.071845+0000","last_active":"2024-09-14T06:28:16.071845+0000","last_peered":"2024-09-14T06:28:16.071845+0000","last_clean":"2024-09-14T06:28:16.071845+0000","last_became_active":"2024-09-14T06:28:14.066720+0000","last_became_peered":"2024-09-14T06:28:14.066720+0000","last_unstale":"2024-09-14T06:28:16.071845+0000","last_undegraded":"2024-09-14T06:28:16.071845+0000","last_fullsized":"2024-09-14T06:28:16.071845+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:18.865038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078953200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.072384+0000","last_change":"2024-09-14T06:28:16.072384+0000","last_active":"2024-09-14T06:28:16.072384+0000","last_peered":"2024-09-14T06:28:16.072384+0000","last_clean":"2024-09-14T06:28:16.072384+0000","last_became_active":"2024-09-14T06:28:14.067568+0000","last_became_peered":"2024-09-14T06:28:14.067568+0000","last_unstale":"2024-09-14T06:28:16.072384+0000","last_undegraded":"2024-09-14T06:28:16.072384+0000","last_fullsized":"2024-09-14T06:28:16.072384+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:25:25.337512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.072011+0000","last_change":"2024-09-14T06:28:16.072011+0000","last_active":"2024-09-14T06:28:16.072011+0000","last_peered":"2024-09-14T06:28:16.072011+0000","last_clean":"2024-09-14T06:28:16.072011+0000","last_became_active":"2024-09-14T06:28:14.065921+0000","last_became_peered":"2024-09-14T06:28:14.065921+0000","last_unstale":"2024-09-14T06:28:16.072011+0000","last_undegraded":"2024-09-14T06:28:16.072011+0000","last_fullsized":"2024-09-14T06:28:16.072011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:45:01.691891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092469000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.072360+0000","last_change":"2024-09-14T06:28:16.072360+0000","last_active":"2024-09-14T06:28:16.072360+0000","last_peered":"2024-09-14T06:28:16.072360+0000","last_clean":"2024-09-14T06:28:16.072360+0000","last_became_active":"2024-09-14T06:28:14.067522+0000","last_became_peered":"2024-09-14T06:28:14.067522+0000","last_unstale":"2024-09-14T06:28:16.072360+0000","last_undegraded":"2024-09-14T06:28:16.072360+0000","last_fullsized":"2024-09-14T06:28:16.072360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:56.670133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056764,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.070458+0000","last_change":"2024-09-14T06:28:16.069659+0000","last_active":"2024-09-14T06:28:16.070458+0000","last_peered":"2024-09-14T06:28:16.070458+0000","last_clean":"2024-09-14T06:28:16.070458+0000","last_became_active":"2024-09-14T06:28:14.065736+0000","last_became_peered":"2024-09-14T06:28:14.065736+0000","last_unstale":"2024-09-14T06:28:16.070458+0000","last_undegraded":"2024-09-14T06:28:16.070458+0000","last_fullsized":"2024-09-14T06:28:16.070458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:29.974020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010548370000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.066424+0000","last_change":"2024-09-14T06:28:16.066424+0000","last_active":"2024-09-14T06:28:16.066424+0000","last_peered":"2024-09-14T06:28:16.066424+0000","last_clean":"2024-09-14T06:28:16.066424+0000","last_became_active":"2024-09-14T06:28:14.065677+0000","last_became_peered":"2024-09-14T06:28:14.065677+0000","last_unstale":"2024-09-14T06:28:16.066424+0000","last_undegraded":"2024-09-14T06:28:16.066424+0000","last_fullsized":"2024-09-14T06:28:16.066424+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:36.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047781599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.066327+0000","last_change":"2024-09-14T06:28:16.066327+0000","last_active":"2024-09-14T06:28:16.066327+0000","last_peered":"2024-09-14T06:28:16.066327+0000","last_clean":"2024-09-14T06:28:16.066327+0000","last_became_active":"2024-09-14T06:28:14.067147+0000","last_became_peered":"2024-09-14T06:28:14.067147+0000","last_unstale":"2024-09-14T06:28:16.066327+0000","last_undegraded":"2024-09-14T06:28:16.066327+0000","last_fullsized":"2024-09-14T06:28:16.066327+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:19:13.309078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042511200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.071663+0000","last_change":"2024-09-14T06:28:16.071663+0000","last_active":"2024-09-14T06:28:16.071663+0000","last_peered":"2024-09-14T06:28:16.071663+0000","last_clean":"2024-09-14T06:28:16.071663+0000","last_became_active":"2024-09-14T06:28:14.065753+0000","last_became_peered":"2024-09-14T06:28:14.065753+0000","last_unstale":"2024-09-14T06:28:16.071663+0000","last_undegraded":"2024-09-14T06:28:16.071663+0000","last_fullsized":"2024-09-14T06:28:16.071663+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:38:56.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066619800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-14T06:28:16.071108+0000","last_change":"2024-09-14T06:28:10.140562+0000","last_active":"2024-09-14T06:28:16.071108+0000","last_peered":"2024-09-14T06:28:16.071108+0000","last_clean":"2024-09-14T06:28:16.071108+0000","last_became_active":"2024-09-14T06:28:10.139552+0000","last_became_peered":"2024-09-14T06:28:10.139552+0000","last_unstale":"2024-09-14T06:28:16.071108+0000","last_undegraded":"2024-09-14T06:28:16.071108+0000","last_fullsized":"2024-09-14T06:28:16.071108+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_clean_scrub_stamp":"2024-09-14T06:28:09.039132+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:00:29.799410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:28:19.960 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-14T06:28:19.960 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-14T06:28:19.960 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-14T06:28:19.960 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-14T06:28:20.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:20.292 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T06:28:20.304 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-14T06:28:20.304 INFO:teuthology.run_tasks:Running task ceph... 2024-09-14T06:28:20.314 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-14T06:28:20.314 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-14T06:28:20.314 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380', 'branch': 'main', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900380', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 950, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5839, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'sleep_before_teardown': 0, 'subset': '99/128', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/1YN9cdEByEgxDm3lqN2v8GTzP/mHK4T/7oMWEBifSOh9bmZZXO3sHGPVW1dKEYnMLNg2KODvzIA8Wq6ZWRz0=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+vaI4OBZX+ycCfdIr6NI27KSkRiUV6FBKOMObmKr/VHKf+FKu+UhLydleKQ0YcZ1uFNsOnPH7i/3IMTt+oxTw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_20:16:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-14T06:28:20.315 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-14T06:28:20.318 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-14T06:28:20.337 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T06:28:20.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T06:28:20.407 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:20.407 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T06:28:20.465 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T06:28:20.465 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-09-14 06:27:37.078246070 +0000 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-09-14 06:24:48.323023536 +0000 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-09-14 06:24:48.323023536 +0000 2024-09-14T06:28:20.523 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-09-14 06:24:48.323023536 +0000 2024-09-14T06:28:20.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T06:28:20.589 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-09-14T06:28:20.589 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-09-14T06:28:20.589 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.2421e-05 s, 5.5 MB/s 2024-09-14T06:28:20.590 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T06:28:20.648 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-09-14 06:27:37.079246047 +0000 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-09-14 06:24:48.651015917 +0000 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-09-14 06:24:48.651015917 +0000 2024-09-14T06:28:20.705 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-09-14 06:24:48.651015917 +0000 2024-09-14T06:28:20.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T06:28:20.769 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-09-14T06:28:20.769 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-09-14T06:28:20.769 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000106138 s, 4.8 MB/s 2024-09-14T06:28:20.770 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T06:28:20.826 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-09-14T06:28:20.881 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-09-14 06:27:37.079246047 +0000 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-09-14 06:24:48.981008251 +0000 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-09-14 06:24:48.981008251 +0000 2024-09-14T06:28:20.882 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-09-14 06:24:48.981008251 +0000 2024-09-14T06:28:20.882 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T06:28:20.945 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-09-14T06:28:20.946 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-09-14T06:28:20.946 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000106961 s, 4.8 MB/s 2024-09-14T06:28:20.947 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T06:28:21.003 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-09-14T06:28:21.059 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T06:28:21.059 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T06:28:21.059 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-14T06:28:21.059 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T06:28:21.059 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T06:28:21.059 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-09-14 06:27:37.079246047 +0000 2024-09-14T06:28:21.060 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-09-14 06:24:49.296000933 +0000 2024-09-14T06:28:21.060 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-09-14 06:24:49.296000933 +0000 2024-09-14T06:28:21.060 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-09-14 06:24:49.296000933 +0000 2024-09-14T06:28:21.060 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T06:28:21.122 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-09-14T06:28:21.123 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-09-14T06:28:21.123 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.0094e-05 s, 5.7 MB/s 2024-09-14T06:28:21.124 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T06:28:21.180 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:28:21.181 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T06:28:21.181 INFO:tasks.ceph:Generating config... 2024-09-14T06:28:21.182 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-14T06:28:21.182 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-14T06:28:21.182 INFO:tasks.ceph:[client] log to stderr = False 2024-09-14T06:28:21.182 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.183 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.184 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.184 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.184 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-14T06:28:21.184 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-14T06:28:21.184 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-14T06:28:21.184 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-14T06:28:21.184 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-14T06:28:21.184 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-14T06:28:21.184 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-14T06:28:21.184 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-14T06:28:21.184 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-14T06:28:21.184 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-14T06:28:21.184 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-14T06:28:21.185 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-14T06:28:21.185 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-14T06:28:21.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-14T06:28:21.262 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.keyring 2024-09-14T06:28:21.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-14T06:28:21.349 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T06:28:21.414 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.170')] 2024-09-14T06:28:21.414 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.170', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-14T06:28:21.415 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:21.415 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T06:28:21.469 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.170 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: generated fsid a7356329-2cae-44e1-a06f-290f5fd22015 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:setting min_mon_release = quincy 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:epoch 0 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:fsid a7356329-2cae-44e1-a06f-290f5fd22015 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:last_changed 2024-09-14T06:28:21.549470+0000 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:created 2024-09-14T06:28:21.549470+0000 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:min_mon_release 17 (quincy) 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:election_strategy: 1 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2024-09-14T06:28:21.550 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-14T06:28:21.551 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T06:28:21.607 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID a7356329-2cae-44e1-a06f-290f5fd22015... 2024-09-14T06:28:21.608 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-14T06:28:21.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-14T06:28:21.661 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-09-14T06:28:21.661 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-09-14T06:28:21.661 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-09-14T06:28:21.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-09-14T06:28:21.663 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-09-14T06:28:21.664 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = 172.21.15.170 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on pool no app = False 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 5000 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = a7356329-2cae-44e1-a06f-290f5fd22015 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-09-14T06:28:21.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-09-14T06:28:21.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1/20 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1/20 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-14T06:28:21.667 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 4/10 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: osd memory target = 1610612736 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:28:21.668 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:28:21.669 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T06:28:21.670 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd = 20 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd_mirror = 15 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: log to stderr = False 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: rbd default features = 125 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.0] 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.671 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.1] 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.2] 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.3] 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.4] 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.5] 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.672 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.673 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.6] 2024-09-14T06:28:21.673 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.673 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.690 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2024-09-14T06:28:21.690 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2024-09-14T06:28:21.690 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T06:28:21.690 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2024-09-14T06:28:21.691 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2024-09-14T06:28:21.692 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.693 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = 172.21.15.170 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on pool no app = False 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 5000 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = a7356329-2cae-44e1-a06f-290f5fd22015 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2024-09-14T06:28:21.694 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2024-09-14T06:28:21.695 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2024-09-14T06:28:21.696 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: bdev async discard = True 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: bdev enable discard = True 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 1/20 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 1/20 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 4/10 2024-09-14T06:28:21.697 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: osd memory target = 1610612736 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:28:21.698 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.699 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:21.700 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd = 20 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: debug rbd_mirror = 15 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: log to stderr = False 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: rbd default features = 125 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.701 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.4] 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.5] 2024-09-14T06:28:21.702 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.703 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.703 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.6] 2024-09-14T06:28:21.703 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T06:28:21.703 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T06:28:21.704 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-14T06:28:21.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-14T06:28:21.800 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-14T06:28:21.801 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:21.801 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-14T06:28:21.816 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:21.816 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-14T06:28:21.871 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T06:28:21.871 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:28:21.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-14T06:28:21.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T06:28:21.914 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:28:21.914 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:21.973 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:28:21.973 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:21.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-14T06:28:21.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T06:28:22.009 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:22.009 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:22.065 INFO:tasks.ceph:Setting up mon nodes... 2024-09-14T06:28:22.065 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-14T06:28:22.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-14T06:28:22.125 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-14T06:28:22.128 INFO:tasks.ceph:Setting up mds nodes... 2024-09-14T06:28:22.128 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-14T06:28:22.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-14T06:28:22.173 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-14T06:28:22.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-14T06:28:22.227 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-14T06:28:22.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-14T06:28:22.321 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-14T06:28:22.333 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-14T06:28:22.414 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-14T06:28:22.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-14T06:28:22.507 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-14T06:28:22.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-14T06:28:22.601 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-14T06:28:22.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-14T06:28:22.694 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-14T06:28:22.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-14T06:28:22.788 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-14T06:28:22.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-14T06:28:22.882 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-14T06:28:22.894 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-14T06:28:22.895 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T06:28:22.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-14T06:28:22.959 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:28:22.960 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-14T06:28:22.960 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:28:22.960 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T06:28:23.024 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T06:28:23.056 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-09-14T06:28:23.058 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-09-14T06:28:23.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-14T06:28:23.127 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-14T06:28:23.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-14T06:28:23.262 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:28:23.262 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-14T06:28:23.262 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:28:23.262 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T06:28:23.326 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T06:28:23.327 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T06:28:23.359 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-09-14T06:28:23.361 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-09-14T06:28:23.361 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-14T06:28:23.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-14T06:28:23.505 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-14T06:28:23.571 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T06:28:23.571 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-14T06:28:23.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:28:23.571 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-14T06:28:23.635 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T06:28:23.635 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T06:28:23.635 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T06:28:23.636 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T06:28:23.636 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T06:28:23.636 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-09-14T06:28:23.636 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T06:28:23.636 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T06:28:23.636 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T06:28:23.636 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T06:28:23.668 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-09-14T06:28:23.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-09-14T06:28:23.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-14T06:28:23.738 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-14T06:28:23.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:23.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:23.904+0000 7f4aaa619640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-14T06:28:23.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:23.905+0000 7f4aaa619640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-14T06:28:23.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:23.905+0000 7f4aaa619640 -1 bdev(0x55a4d945d800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-14T06:28:23.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:23.905+0000 7f4aaa619640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-14T06:28:27.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-14T06:28:27.583 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:27.673 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:27.672+0000 7fc05eed6640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-14T06:28:27.673 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:27.672+0000 7fc05eed6640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-14T06:28:27.673 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:27.672+0000 7fc05eed6640 -1 bdev(0x5577ce559800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-14T06:28:27.674 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:27.672+0000 7fc05eed6640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-14T06:28:31.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-14T06:28:31.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:31.202 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:31.200+0000 7f8d4a41a640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-14T06:28:31.202 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:31.201+0000 7f8d4a41a640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-14T06:28:31.203 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:31.201+0000 7f8d4a41a640 -1 bdev(0x560c17235800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-14T06:28:31.203 INFO:teuthology.orchestra.run.smithi170.stderr:2024-09-14T06:28:31.201+0000 7f8d4a41a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-14T06:28:34.100 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-14T06:28:34.127 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-14T06:28:34.127 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:28:34.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-14T06:28:34.159 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.159 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-14T06:28:34.194 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-14T06:28:34.261 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-14T06:28:34.327 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-14T06:28:34.328 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-14T06:28:34.347 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.347 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-14T06:28:34.384 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.385 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-14T06:28:34.441 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.441 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-14T06:28:34.497 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.497 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-14T06:28:34.553 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.553 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-14T06:28:34.608 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.608 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-14T06:28:34.663 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.664 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-14T06:28:34.719 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:34.719 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-14T06:28:34.774 INFO:tasks.ceph:Adding keys to all mons... 2024-09-14T06:28:34.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-14T06:28:34.837 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2024-09-14T06:28:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmhE58BxAAlCXzbJulwpHILbIdUh2Xog== 2024-09-14T06:28:34.837 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2024-09-14T06:28:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAHLeVmhXr+NRAAOE+m1L5Bwg0M7uzWPEEl3Q== 2024-09-14T06:28:34.837 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2024-09-14T06:28:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQALLeVmPDceKBAA2ACPBT+GWuk4yacaSB0Yug== 2024-09-14T06:28:34.837 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAPLeVmNiUIDBAAwKcNQmbjErbByAUv8XIAqA== 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmefFaChAAV9NDYINk4Aw382szBRZLCQ== 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror] 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmAtGIDRAA+wFSz8WbevNqt9bWVd/kuw== 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVm6NEcExAACDp1oh4T6v9eXG+gV3ZjWQ== 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmVIGzGBAAf/D/eHi95DEpzdYw6f381g== 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVm0pw/HhAAj+52xvAEPgzhlM1D0sEYUQ== 2024-09-14T06:28:34.838 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2024-09-14T06:28:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmYJ3QIxAAAqj9s+XRIQTN5+UftoIjxw== 2024-09-14T06:28:34.839 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.4] 2024-09-14T06:28:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmvxNcKRAA/QsymQ2p1drXlJUtAwbvSg== 2024-09-14T06:28:34.839 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.5] 2024-09-14T06:28:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVm+Sf9LhAAnK3TTFTfDL9IMiCk48c9XQ== 2024-09-14T06:28:34.839 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.6] 2024-09-14T06:28:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmRliQNBAAgdrICawIAVnMjGsndd1+OQ== 2024-09-14T06:28:34.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T06:28:34.931 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T06:28:34.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T06:28:35.033 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T06:28:35.084 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.138 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T06:28:35.571 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-14T06:28:35.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-14T06:28:35.596 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-14T06:28:35.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-14T06:28:35.717 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T06:28:35.772 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-14T06:28:35.773 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-14T06:28:35.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-14T06:28:35.815 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-14T06:28:35.815 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-14T06:28:35.815 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-14T06:28:35.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-14T06:28:35.818 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-14T06:28:35.818 DEBUG:tasks.ceph:set 0 configs 2024-09-14T06:28:35.818 INFO:tasks.ceph:Setting crush tunables to default 2024-09-14T06:28:35.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-14T06:28:35.988 INFO:teuthology.orchestra.run.smithi170.stderr:adjusted tunables profile to default 2024-09-14T06:28:35.999 INFO:tasks.ceph:check_enable_crimson: False 2024-09-14T06:28:35.999 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-14T06:28:35.999 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:35.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-14T06:28:36.024 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:36.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-14T06:28:36.089 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:28:36.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-14T06:28:36.154 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd new 7de0430d-00ac-4a23-b678-2d3db13c7c08 0 2024-09-14T06:28:36.368 INFO:teuthology.orchestra.run.smithi170.stdout:0 2024-09-14T06:28:36.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd new 2ae5c1c1-21e0-47aa-ba83-adc731863719 1 2024-09-14T06:28:36.552 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-09-14T06:28:36.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd new eab8881a-6d1e-495c-bba6-27c1ac2b98d9 2 2024-09-14T06:28:36.736 INFO:teuthology.orchestra.run.smithi170.stdout:2 2024-09-14T06:28:36.747 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-14T06:28:36.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-14T06:28:36.749 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-14T06:28:36.750 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-14T06:28:36.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-14T06:28:36.752 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-14T06:28:36.752 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-14T06:28:36.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-14T06:28:36.754 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-14T06:28:36.754 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T06:28:36.926 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:36.926 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":5,"fsid":"a7356329-2cae-44e1-a06f-290f5fd22015","created":"2024-09-14T06:28:35.900384+0000","modified":"2024-09-14T06:28:36.735545+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T06:28:36.735545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7de0430d-00ac-4a23-b678-2d3db13c7c08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2ae5c1c1-21e0-47aa-ba83-adc731863719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eab8881a-6d1e-495c-bba6-27c1ac2b98d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:36.937 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-14T06:28:36.937 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-14T06:28:37.239 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T06:28:37.397 INFO:teuthology.misc.health.smithi170.stdout: 2024-09-14T06:28:37.397 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":5,"fsid":"a7356329-2cae-44e1-a06f-290f5fd22015","created":"2024-09-14T06:28:35.900384+0000","modified":"2024-09-14T06:28:36.735545+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T06:28:36.735545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7de0430d-00ac-4a23-b678-2d3db13c7c08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2ae5c1c1-21e0-47aa-ba83-adc731863719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eab8881a-6d1e-495c-bba6-27c1ac2b98d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:37.406 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T06:28:37.956 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:28:37.955+0000 7f568c72c640 -1 Falling back to public interface 2024-09-14T06:28:38.179 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:28:38.178+0000 7f0c79c14640 -1 Falling back to public interface 2024-09-14T06:28:38.443 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:28:38.441+0000 7ff0a2820640 -1 Falling back to public interface 2024-09-14T06:28:41.991 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:28:41.989+0000 7ff0a2820640 -1 osd.2 0 log_to_monitors true 2024-09-14T06:28:41.996 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:28:41.995+0000 7f0c79c14640 -1 osd.1 0 log_to_monitors true 2024-09-14T06:28:41.997 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:28:41.996+0000 7f568c72c640 -1 osd.0 0 log_to_monitors true 2024-09-14T06:28:42.683 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-14T06:28:42.681+0000 7f27b01d0640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-14T06:28:43.710 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T06:28:44.085 INFO:teuthology.misc.health.smithi170.stdout: 2024-09-14T06:28:44.086 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":7,"fsid":"a7356329-2cae-44e1-a06f-290f5fd22015","created":"2024-09-14T06:28:35.900384+0000","modified":"2024-09-14T06:28:43.526336+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T06:28:36.735545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7de0430d-00ac-4a23-b678-2d3db13c7c08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2ae5c1c1-21e0-47aa-ba83-adc731863719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eab8881a-6d1e-495c-bba6-27c1ac2b98d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:44.101 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T06:28:44.753 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:28:44.751+0000 7f56886bb640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T06:28:44.769 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:28:44.768+0000 7f56804da640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T06:28:44.772 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:28:44.770+0000 7ff09e925640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T06:28:44.780 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:28:44.779+0000 7f0c75624640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T06:28:44.787 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:28:44.786+0000 7ff095f30640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T06:28:44.799 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:28:44.798+0000 7f0c6d443640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T06:28:50.404 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T06:28:50.689 INFO:teuthology.misc.health.smithi170.stdout: 2024-09-14T06:28:50.689 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":12,"fsid":"a7356329-2cae-44e1-a06f-290f5fd22015","created":"2024-09-14T06:28:35.900384+0000","modified":"2024-09-14T06:28:49.760828+0000","last_up_change":"2024-09-14T06:28:45.753509+0000","last_in_change":"2024-09-14T06:28:36.735545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:47.496517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7de0430d-00ac-4a23-b678-2d3db13c7c08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6801","nonce":4002621419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6803","nonce":4002621419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6807","nonce":4002621419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6805","nonce":4002621419}]},"public_addr":"172.21.15.170:6801/4002621419","cluster_addr":"172.21.15.170:6803/4002621419","heartbeat_back_addr":"172.21.15.170:6807/4002621419","heartbeat_front_addr":"172.21.15.170:6805/4002621419","state":["exists","up"]},{"osd":1,"uuid":"2ae5c1c1-21e0-47aa-ba83-adc731863719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6809","nonce":3258460295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6811","nonce":3258460295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6815","nonce":3258460295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6813","nonce":3258460295}]},"public_addr":"172.21.15.170:6809/3258460295","cluster_addr":"172.21.15.170:6811/3258460295","heartbeat_back_addr":"172.21.15.170:6815/3258460295","heartbeat_front_addr":"172.21.15.170:6813/3258460295","state":["exists","up"]},{"osd":2,"uuid":"eab8881a-6d1e-495c-bba6-27c1ac2b98d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6817","nonce":2319540746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6819","nonce":2319540746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6823","nonce":2319540746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6821","nonce":2319540746}]},"public_addr":"172.21.15.170:6817/2319540746","cluster_addr":"172.21.15.170:6819/2319540746","heartbeat_back_addr":"172.21.15.170:6823/2319540746","heartbeat_front_addr":"172.21.15.170:6821/2319540746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.007661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:42.984623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.008465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:50.700 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-14T06:28:50.700 INFO:tasks.ceph:Creating RBD pool 2024-09-14T06:28:50.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-14T06:28:51.767 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'rbd' created 2024-09-14T06:28:51.782 DEBUG:teuthology.orchestra.run.smithi170:> rbd --cluster cluster2 pool init rbd 2024-09-14T06:28:54.787 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-14T06:28:54.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 config log 1 --format=json 2024-09-14T06:28:54.788 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-14T06:28:55.080 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:55.092 INFO:teuthology.orchestra.run.smithi170.stdout:[{"version":4,"timestamp":"2024-09-14T06:28:44.780131+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"5056.702593"}]}] 2024-09-14T06:28:55.093 INFO:tasks.ceph_manager:config epoch is 4 2024-09-14T06:28:55.093 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-14T06:28:55.093 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-14T06:28:55.093 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-14T06:28:55.421 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:55.433 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":2265476612},{"type":"v1","addr":"172.21.15.1:6827","nonce":2265476612}]},"active_addr":"172.21.15.1:6827/2265476612","active_change":"2024-09-14T06:28:41.472853+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":560885498}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":900759951}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3775917641}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3561311299}]}]} 2024-09-14T06:28:55.435 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-14T06:28:55.435 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-14T06:28:55.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T06:28:55.714 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:55.714 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":16,"fsid":"a7356329-2cae-44e1-a06f-290f5fd22015","created":"2024-09-14T06:28:35.900384+0000","modified":"2024-09-14T06:28:54.773083+0000","last_up_change":"2024-09-14T06:28:45.753509+0000","last_in_change":"2024-09-14T06:28:36.735545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:47.496517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T06:28:50.986399+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7de0430d-00ac-4a23-b678-2d3db13c7c08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6801","nonce":4002621419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6803","nonce":4002621419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6807","nonce":4002621419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6805","nonce":4002621419}]},"public_addr":"172.21.15.170:6801/4002621419","cluster_addr":"172.21.15.170:6803/4002621419","heartbeat_back_addr":"172.21.15.170:6807/4002621419","heartbeat_front_addr":"172.21.15.170:6805/4002621419","state":["exists","up"]},{"osd":1,"uuid":"2ae5c1c1-21e0-47aa-ba83-adc731863719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6809","nonce":3258460295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6811","nonce":3258460295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6815","nonce":3258460295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6813","nonce":3258460295}]},"public_addr":"172.21.15.170:6809/3258460295","cluster_addr":"172.21.15.170:6811/3258460295","heartbeat_back_addr":"172.21.15.170:6815/3258460295","heartbeat_front_addr":"172.21.15.170:6813/3258460295","state":["exists","up"]},{"osd":2,"uuid":"eab8881a-6d1e-495c-bba6-27c1ac2b98d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6817","nonce":2319540746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6819","nonce":2319540746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6823","nonce":2319540746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6821","nonce":2319540746}]},"public_addr":"172.21.15.170:6817/2319540746","cluster_addr":"172.21.15.170:6819/2319540746","heartbeat_back_addr":"172.21.15.170:6823/2319540746","heartbeat_front_addr":"172.21.15.170:6821/2319540746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.007661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:42.984623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.008465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:55.727 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-14T06:28:55.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T06:28:56.012 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:56.012 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":16,"fsid":"a7356329-2cae-44e1-a06f-290f5fd22015","created":"2024-09-14T06:28:35.900384+0000","modified":"2024-09-14T06:28:54.773083+0000","last_up_change":"2024-09-14T06:28:45.753509+0000","last_in_change":"2024-09-14T06:28:36.735545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:47.496517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T06:28:50.986399+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7de0430d-00ac-4a23-b678-2d3db13c7c08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6801","nonce":4002621419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6803","nonce":4002621419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6807","nonce":4002621419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6805","nonce":4002621419}]},"public_addr":"172.21.15.170:6801/4002621419","cluster_addr":"172.21.15.170:6803/4002621419","heartbeat_back_addr":"172.21.15.170:6807/4002621419","heartbeat_front_addr":"172.21.15.170:6805/4002621419","state":["exists","up"]},{"osd":1,"uuid":"2ae5c1c1-21e0-47aa-ba83-adc731863719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6809","nonce":3258460295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6811","nonce":3258460295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6815","nonce":3258460295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6813","nonce":3258460295}]},"public_addr":"172.21.15.170:6809/3258460295","cluster_addr":"172.21.15.170:6811/3258460295","heartbeat_back_addr":"172.21.15.170:6815/3258460295","heartbeat_front_addr":"172.21.15.170:6813/3258460295","state":["exists","up"]},{"osd":2,"uuid":"eab8881a-6d1e-495c-bba6-27c1ac2b98d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6817","nonce":2319540746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6819","nonce":2319540746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6823","nonce":2319540746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6821","nonce":2319540746}]},"public_addr":"172.21.15.170:6817/2319540746","cluster_addr":"172.21.15.170:6819/2319540746","heartbeat_back_addr":"172.21.15.170:6823/2319540746","heartbeat_front_addr":"172.21.15.170:6821/2319540746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.007661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:42.984623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.008465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:28:56.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-14T06:28:56.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-14T06:28:56.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-14T06:28:56.164 INFO:teuthology.orchestra.run.smithi170.stdout:34359738374 2024-09-14T06:28:56.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-14T06:28:56.165 INFO:teuthology.orchestra.run.smithi170.stdout:34359738374 2024-09-14T06:28:56.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-14T06:28:56.166 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2024-09-14T06:28:56.167 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-14T06:28:56.505 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2024-09-14T06:28:56.507 INFO:teuthology.orchestra.run.smithi170.stdout:34359738372 2024-09-14T06:28:56.514 INFO:teuthology.orchestra.run.smithi170.stdout:34359738373 2024-09-14T06:28:56.520 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-09-14T06:28:56.521 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-09-14T06:28:56.527 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-09-14T06:28:57.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-14T06:28:57.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-14T06:28:57.528 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-14T06:28:57.838 INFO:teuthology.orchestra.run.smithi170.stdout:34359738375 2024-09-14T06:28:57.852 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.1 2024-09-14T06:28:57.852 DEBUG:teuthology.parallel:result is None 2024-09-14T06:28:57.882 INFO:teuthology.orchestra.run.smithi170.stdout:34359738374 2024-09-14T06:28:57.895 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.0 2024-09-14T06:28:57.895 DEBUG:teuthology.parallel:result is None 2024-09-14T06:28:57.911 INFO:teuthology.orchestra.run.smithi170.stdout:34359738375 2024-09-14T06:28:57.923 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.2 2024-09-14T06:28:57.924 DEBUG:teuthology.parallel:result is None 2024-09-14T06:28:57.924 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-14T06:28:57.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T06:28:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:58.247 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-09-14T06:28:58.259 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-14T06:28:57.480631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95160,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020360,"statfs":{"total":289910292480,"available":289812848640,"internally_reserved":0,"allocated":1613824,"data_stored":1303162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.713476"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782642+0000","last_change":"2024-09-14T06:28:54.782642+0000","last_active":"2024-09-14T06:28:54.782642+0000","last_peered":"2024-09-14T06:28:54.782642+0000","last_clean":"2024-09-14T06:28:54.782642+0000","last_became_active":"2024-09-14T06:28:52.776292+0000","last_became_peered":"2024-09-14T06:28:52.776292+0000","last_unstale":"2024-09-14T06:28:54.782642+0000","last_undegraded":"2024-09-14T06:28:54.782642+0000","last_fullsized":"2024-09-14T06:28:54.782642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:48:57.788740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010453089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782290+0000","last_change":"2024-09-14T06:28:54.782290+0000","last_active":"2024-09-14T06:28:54.782290+0000","last_peered":"2024-09-14T06:28:54.782290+0000","last_clean":"2024-09-14T06:28:54.782290+0000","last_became_active":"2024-09-14T06:28:52.776371+0000","last_became_peered":"2024-09-14T06:28:52.776371+0000","last_unstale":"2024-09-14T06:28:54.782290+0000","last_undegraded":"2024-09-14T06:28:54.782290+0000","last_fullsized":"2024-09-14T06:28:54.782290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:49:06.151379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077233399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782567+0000","last_change":"2024-09-14T06:28:54.782567+0000","last_active":"2024-09-14T06:28:54.782567+0000","last_peered":"2024-09-14T06:28:54.782567+0000","last_clean":"2024-09-14T06:28:54.782567+0000","last_became_active":"2024-09-14T06:28:52.776250+0000","last_became_peered":"2024-09-14T06:28:52.776250+0000","last_unstale":"2024-09-14T06:28:54.782567+0000","last_undegraded":"2024-09-14T06:28:54.782567+0000","last_fullsized":"2024-09-14T06:28:54.782567+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:21:07.454036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011012839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.781991+0000","last_change":"2024-09-14T06:28:54.781991+0000","last_active":"2024-09-14T06:28:54.781991+0000","last_peered":"2024-09-14T06:28:54.781991+0000","last_clean":"2024-09-14T06:28:54.781991+0000","last_became_active":"2024-09-14T06:28:52.776740+0000","last_became_peered":"2024-09-14T06:28:52.776740+0000","last_unstale":"2024-09-14T06:28:54.781991+0000","last_undegraded":"2024-09-14T06:28:54.781991+0000","last_fullsized":"2024-09-14T06:28:54.781991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:17:00.207670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064401499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.780821+0000","last_change":"2024-09-14T06:28:54.780145+0000","last_active":"2024-09-14T06:28:54.780821+0000","last_peered":"2024-09-14T06:28:54.780821+0000","last_clean":"2024-09-14T06:28:54.780821+0000","last_became_active":"2024-09-14T06:28:52.776357+0000","last_became_peered":"2024-09-14T06:28:52.776357+0000","last_unstale":"2024-09-14T06:28:54.780821+0000","last_undegraded":"2024-09-14T06:28:54.780821+0000","last_fullsized":"2024-09-14T06:28:54.780821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:57:29.100954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096998400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.777770+0000","last_change":"2024-09-14T06:28:54.777770+0000","last_active":"2024-09-14T06:28:54.777770+0000","last_peered":"2024-09-14T06:28:54.777770+0000","last_clean":"2024-09-14T06:28:54.777770+0000","last_became_active":"2024-09-14T06:28:52.777933+0000","last_became_peered":"2024-09-14T06:28:52.777933+0000","last_unstale":"2024-09-14T06:28:54.777770+0000","last_undegraded":"2024-09-14T06:28:54.777770+0000","last_fullsized":"2024-09-14T06:28:54.777770+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:50.657246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049603799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.777772+0000","last_change":"2024-09-14T06:28:54.777772+0000","last_active":"2024-09-14T06:28:54.777772+0000","last_peered":"2024-09-14T06:28:54.777772+0000","last_clean":"2024-09-14T06:28:54.777772+0000","last_became_active":"2024-09-14T06:28:52.777950+0000","last_became_peered":"2024-09-14T06:28:52.777950+0000","last_unstale":"2024-09-14T06:28:54.777772+0000","last_undegraded":"2024-09-14T06:28:54.777772+0000","last_fullsized":"2024-09-14T06:28:54.777772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:44:17.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504179,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782038+0000","last_change":"2024-09-14T06:28:54.782038+0000","last_active":"2024-09-14T06:28:54.782038+0000","last_peered":"2024-09-14T06:28:54.782038+0000","last_clean":"2024-09-14T06:28:54.782038+0000","last_became_active":"2024-09-14T06:28:52.775997+0000","last_became_peered":"2024-09-14T06:28:52.775997+0000","last_unstale":"2024-09-14T06:28:54.782038+0000","last_undegraded":"2024-09-14T06:28:54.782038+0000","last_fullsized":"2024-09-14T06:28:54.782038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:48.639577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000668876,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.781277+0000","last_change":"2024-09-14T06:28:48.763379+0000","last_active":"2024-09-14T06:28:54.781277+0000","last_peered":"2024-09-14T06:28:54.781277+0000","last_clean":"2024-09-14T06:28:54.781277+0000","last_became_active":"2024-09-14T06:28:48.763050+0000","last_became_peered":"2024-09-14T06:28:48.763050+0000","last_unstale":"2024-09-14T06:28:54.781277+0000","last_undegraded":"2024-09-14T06:28:54.781277+0000","last_fullsized":"2024-09-14T06:28:54.781277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_clean_scrub_stamp":"2024-09-14T06:28:47.755018+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:01:18.238865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31916,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":737280,"data_stored":631183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31916,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":737280,"data_stored":631183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:28:58.260 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T06:28:58.541 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:58.542 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-09-14T06:28:58.553 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-14T06:28:57.480631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95160,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020360,"statfs":{"total":289910292480,"available":289812848640,"internally_reserved":0,"allocated":1613824,"data_stored":1303162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.713476"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782642+0000","last_change":"2024-09-14T06:28:54.782642+0000","last_active":"2024-09-14T06:28:54.782642+0000","last_peered":"2024-09-14T06:28:54.782642+0000","last_clean":"2024-09-14T06:28:54.782642+0000","last_became_active":"2024-09-14T06:28:52.776292+0000","last_became_peered":"2024-09-14T06:28:52.776292+0000","last_unstale":"2024-09-14T06:28:54.782642+0000","last_undegraded":"2024-09-14T06:28:54.782642+0000","last_fullsized":"2024-09-14T06:28:54.782642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:48:57.788740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010453089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782290+0000","last_change":"2024-09-14T06:28:54.782290+0000","last_active":"2024-09-14T06:28:54.782290+0000","last_peered":"2024-09-14T06:28:54.782290+0000","last_clean":"2024-09-14T06:28:54.782290+0000","last_became_active":"2024-09-14T06:28:52.776371+0000","last_became_peered":"2024-09-14T06:28:52.776371+0000","last_unstale":"2024-09-14T06:28:54.782290+0000","last_undegraded":"2024-09-14T06:28:54.782290+0000","last_fullsized":"2024-09-14T06:28:54.782290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:49:06.151379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077233399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782567+0000","last_change":"2024-09-14T06:28:54.782567+0000","last_active":"2024-09-14T06:28:54.782567+0000","last_peered":"2024-09-14T06:28:54.782567+0000","last_clean":"2024-09-14T06:28:54.782567+0000","last_became_active":"2024-09-14T06:28:52.776250+0000","last_became_peered":"2024-09-14T06:28:52.776250+0000","last_unstale":"2024-09-14T06:28:54.782567+0000","last_undegraded":"2024-09-14T06:28:54.782567+0000","last_fullsized":"2024-09-14T06:28:54.782567+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:21:07.454036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011012839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.781991+0000","last_change":"2024-09-14T06:28:54.781991+0000","last_active":"2024-09-14T06:28:54.781991+0000","last_peered":"2024-09-14T06:28:54.781991+0000","last_clean":"2024-09-14T06:28:54.781991+0000","last_became_active":"2024-09-14T06:28:52.776740+0000","last_became_peered":"2024-09-14T06:28:52.776740+0000","last_unstale":"2024-09-14T06:28:54.781991+0000","last_undegraded":"2024-09-14T06:28:54.781991+0000","last_fullsized":"2024-09-14T06:28:54.781991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:17:00.207670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064401499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.780821+0000","last_change":"2024-09-14T06:28:54.780145+0000","last_active":"2024-09-14T06:28:54.780821+0000","last_peered":"2024-09-14T06:28:54.780821+0000","last_clean":"2024-09-14T06:28:54.780821+0000","last_became_active":"2024-09-14T06:28:52.776357+0000","last_became_peered":"2024-09-14T06:28:52.776357+0000","last_unstale":"2024-09-14T06:28:54.780821+0000","last_undegraded":"2024-09-14T06:28:54.780821+0000","last_fullsized":"2024-09-14T06:28:54.780821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:57:29.100954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096998400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.777770+0000","last_change":"2024-09-14T06:28:54.777770+0000","last_active":"2024-09-14T06:28:54.777770+0000","last_peered":"2024-09-14T06:28:54.777770+0000","last_clean":"2024-09-14T06:28:54.777770+0000","last_became_active":"2024-09-14T06:28:52.777933+0000","last_became_peered":"2024-09-14T06:28:52.777933+0000","last_unstale":"2024-09-14T06:28:54.777770+0000","last_undegraded":"2024-09-14T06:28:54.777770+0000","last_fullsized":"2024-09-14T06:28:54.777770+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:50.657246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049603799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.777772+0000","last_change":"2024-09-14T06:28:54.777772+0000","last_active":"2024-09-14T06:28:54.777772+0000","last_peered":"2024-09-14T06:28:54.777772+0000","last_clean":"2024-09-14T06:28:54.777772+0000","last_became_active":"2024-09-14T06:28:52.777950+0000","last_became_peered":"2024-09-14T06:28:52.777950+0000","last_unstale":"2024-09-14T06:28:54.777772+0000","last_undegraded":"2024-09-14T06:28:54.777772+0000","last_fullsized":"2024-09-14T06:28:54.777772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:44:17.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504179,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.782038+0000","last_change":"2024-09-14T06:28:54.782038+0000","last_active":"2024-09-14T06:28:54.782038+0000","last_peered":"2024-09-14T06:28:54.782038+0000","last_clean":"2024-09-14T06:28:54.782038+0000","last_became_active":"2024-09-14T06:28:52.775997+0000","last_became_peered":"2024-09-14T06:28:52.775997+0000","last_unstale":"2024-09-14T06:28:54.782038+0000","last_undegraded":"2024-09-14T06:28:54.782038+0000","last_fullsized":"2024-09-14T06:28:54.782038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:48.639577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000668876,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T06:28:54.781277+0000","last_change":"2024-09-14T06:28:48.763379+0000","last_active":"2024-09-14T06:28:54.781277+0000","last_peered":"2024-09-14T06:28:54.781277+0000","last_clean":"2024-09-14T06:28:54.781277+0000","last_became_active":"2024-09-14T06:28:48.763050+0000","last_became_peered":"2024-09-14T06:28:48.763050+0000","last_unstale":"2024-09-14T06:28:54.781277+0000","last_undegraded":"2024-09-14T06:28:54.781277+0000","last_fullsized":"2024-09-14T06:28:54.781277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_clean_scrub_stamp":"2024-09-14T06:28:47.755018+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:01:18.238865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31916,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":737280,"data_stored":631183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31916,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":737280,"data_stored":631183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:28:58.554 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-14T06:28:58.554 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-14T06:28:58.554 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-14T06:28:58.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-14T06:28:58.874 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T06:28:58.874 INFO:teuthology.orchestra.run.smithi170.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T06:28:58.886 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-14T06:28:58.887 INFO:teuthology.run_tasks:Running task exec... 2024-09-14T06:28:58.896 INFO:teuthology.task.exec:Executing custom commands... 2024-09-14T06:28:58.896 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:28:58.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:28:59.258 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.4] 2024-09-14T06:28:59.258 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDgLOVm6yv/MxAAtji6d0LTAhJd4j570uFv9A== 2024-09-14T06:28:59.258 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:28:59.258 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:28:59.258 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.4 2024-09-14T06:28:59.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:28:59.627 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.5] 2024-09-14T06:28:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDgLOVmN3+PORAAWpdNTj3dV5RoEZG7J9/Okw== 2024-09-14T06:28:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:28:59.627 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:28:59.627 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.5 2024-09-14T06:28:59.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:00.000 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.6] 2024-09-14T06:29:00.000 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDhLOVmgCmEAxAAT89M6/A1O61xocQ5XQb94A== 2024-09-14T06:29:00.000 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:00.000 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:00.000 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.6 2024-09-14T06:29:00.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:00.376 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.4] 2024-09-14T06:29:00.376 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmvxNcKRAA/QsymQ2p1drXlJUtAwbvSg== 2024-09-14T06:29:00.376 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:00.376 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:00.376 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.4 2024-09-14T06:29:00.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:00.751 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.5] 2024-09-14T06:29:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVm+Sf9LhAAnK3TTFTfDL9IMiCk48c9XQ== 2024-09-14T06:29:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:00.751 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:00.751 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.5 2024-09-14T06:29:00.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:01.132 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.6] 2024-09-14T06:29:01.132 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmRliQNBAAgdrICawIAVnMjGsndd1+OQ== 2024-09-14T06:29:01.132 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:01.132 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:01.133 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.6 2024-09-14T06:29:01.147 INFO:teuthology.run_tasks:Running task exec... 2024-09-14T06:29:01.156 INFO:teuthology.task.exec:Executing custom commands... 2024-09-14T06:29:01.156 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T06:29:01.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:01.518 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror] 2024-09-14T06:29:01.519 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDgLOVmvIAsGhAA870jo170wegxMn7oXaukig== 2024-09-14T06:29:01.519 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-14T06:29:01.519 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:01.519 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror 2024-09-14T06:29:01.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:01.891 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2024-09-14T06:29:01.891 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDgLOVm57OXHRAAQU+bf8Y88H87JCgpsVlGFw== 2024-09-14T06:29:01.892 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:01.892 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:01.892 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.0 2024-09-14T06:29:01.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:02.266 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2024-09-14T06:29:02.267 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDgLOVmeMdEIxAAPMZ/fvIM2co2eOJMbZcdgQ== 2024-09-14T06:29:02.267 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:02.267 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:02.267 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.1 2024-09-14T06:29:02.281 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:02.639 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2024-09-14T06:29:02.639 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDgLOVmFTjRKBAAu/+Nn3rD6BwPc6kEFkYVhw== 2024-09-14T06:29:02.639 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:02.639 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:02.640 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.2 2024-09-14T06:29:02.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:03.016 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2024-09-14T06:29:03.016 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDgLOVmETtfLhAAIogfIZo650stp+MZ6429qQ== 2024-09-14T06:29:03.016 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:03.016 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:03.016 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.3 2024-09-14T06:29:03.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:03.391 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror] 2024-09-14T06:29:03.391 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmAtGIDRAA+wFSz8WbevNqt9bWVd/kuw== 2024-09-14T06:29:03.391 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-14T06:29:03.391 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:03.392 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror 2024-09-14T06:29:03.405 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:03.764 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.0] 2024-09-14T06:29:03.764 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVm6NEcExAACDp1oh4T6v9eXG+gV3ZjWQ== 2024-09-14T06:29:03.764 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:03.764 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:03.765 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.0 2024-09-14T06:29:03.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:04.138 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.1] 2024-09-14T06:29:04.138 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmVIGzGBAAf/D/eHi95DEpzdYw6f381g== 2024-09-14T06:29:04.138 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:04.138 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:04.139 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.1 2024-09-14T06:29:04.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:04.513 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.2] 2024-09-14T06:29:04.513 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVm0pw/HhAAj+52xvAEPgzhlM1D0sEYUQ== 2024-09-14T06:29:04.514 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:04.514 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:04.514 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.2 2024-09-14T06:29:04.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T06:29:04.887 INFO:teuthology.orchestra.run.smithi170.stdout:[client.mirror.3] 2024-09-14T06:29:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAGLeVmYJ3QIxAAAqj9s+XRIQTN5+UftoIjxw== 2024-09-14T06:29:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: caps mon = "profile rbd-mirror" 2024-09-14T06:29:04.887 INFO:teuthology.orchestra.run.smithi170.stdout: caps osd = "profile rbd" 2024-09-14T06:29:04.888 INFO:teuthology.orchestra.run.smithi170.stderr:updated caps for client.mirror.3 2024-09-14T06:29:04.901 INFO:teuthology.run_tasks:Running task workunit... 2024-09-14T06:29:04.915 INFO:tasks.workunit:Pulling workunits from ref 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-14T06:29:04.915 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-14T06:29:04.915 DEBUG:teuthology.orchestra.run.smithi170:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T06:29:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:29:04.931 INFO:teuthology.orchestra.run.smithi170.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-14T06:29:04.931 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T06:29:04.986 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T06:29:04.986 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-14T06:29:05.041 INFO:tasks.workunit:timeout=3h 2024-09-14T06:29:05.041 INFO:tasks.workunit:cleanup=True 2024-09-14T06:29:05.042 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-14T06:29:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-14T06:30:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Note: switching to '0d61375b8ab53be4c233f5b93e463edfdff9316a'. 2024-09-14T06:30:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2024-09-14T06:30:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-14T06:30:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:state without impacting any branches by switching back to a branch. 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: git switch -c 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Or undo this operation with: 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: git switch - 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2024-09-14T06:30:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-14T06:30:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: 2024-09-14T06:30:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:HEAD is now at 0d61375b8ab Merge pull request #59485 from mcv21/cephadm_image_warn 2024-09-14T06:30:00.619 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-14T06:30:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-14T06:30:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-14T06:30:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-14T06:30:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-14T06:30:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-14T06:30:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-14T06:30:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-14T06:30:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-14T06:30:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-14T06:30:01.613 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-09-14T06:30:01.613 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-14T06:30:01.629 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-14T06:30:01.630 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-14T06:30:01.631 DEBUG:teuthology.orchestra.run.smithi170:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0d61375b8ab53be4c233f5b93e463edfdff9316a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T06:30:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ IMAGE_COUNT=50 2024-09-14T06:30:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ export LOCKDEP=0 2024-09-14T06:30:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ LOCKDEP=0 2024-09-14T06:30:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T06:30:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-14T06:30:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ RBD_MIRROR_INSTANCES=2 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CLUSTER1=cluster1 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CLUSTER2=cluster2 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ POOL=mirror 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ PARENT_POOL=mirror_parent 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ NS1=ns1 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ NS2=ns2 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ TEMPDIR= 2024-09-14T06:30:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ID=mirror 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ RBD_MIRROR_MODE=journal 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ MIRROR_POOL_MODE=pool 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' journal = snapshot ']' 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS='--id mirror' 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ LAST_MIRROR_INSTANCE=1 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T06:30:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-14T06:30:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_BIN=. 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_SRC=. 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 -gt 0 ']' 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local c 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup_tempdir 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ mktemp -d 2024-09-14T06:30:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ TEMPDIR=/tmp/tmp.RQ7U70XNbJ 2024-09-14T06:30:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z 1 ']' 2024-09-14T06:30:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup_pools cluster1 cluster2 2024-09-14T06:30:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster2 2024-09-14T06:30:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_map_file 2024-09-14T06:30:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_addr 2024-09-14T06:30:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local admin_key_file 2024-09-14T06:30:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid 2024-09-14T06:30:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-14T06:30:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' already exists 2024-09-14T06:30:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-14T06:30:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' already exists 2024-09-14T06:30:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-14T06:30:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-14T06:30:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-09-14T06:30:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-14T06:30:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-14T06:30:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-14T06:30:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-09-14T06:30:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster1 mirror cluster2 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster2 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster2 2024-09-14T06:30:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2024-09-14T06:30:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2024-09-14T06:30:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2024-09-14T06:30:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T06:30:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:30:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=816dc148-8c48-463d-aebc-521431f43101 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror_parent 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster2 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster2 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:30:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=be6e8f05-2f8c-4493-8fe3-3c85bf4ce4e5 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ setup_pools cluster2 cluster1 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster1 2024-09-14T06:30:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_map_file 2024-09-14T06:30:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mon_addr 2024-09-14T06:30:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local admin_key_file 2024-09-14T06:30:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid 2024-09-14T06:30:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-14T06:30:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' already exists 2024-09-14T06:30:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-14T06:30:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' already exists 2024-09-14T06:30:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-14T06:30:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-14T06:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-09-14T06:30:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-14T06:30:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-14T06:30:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-14T06:30:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-09-14T06:30:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-14T06:30:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2024-09-14T06:30:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z ']' 2024-09-14T06:30:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster2 mirror cluster1 2024-09-14T06:30:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster1 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster1 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2024-09-14T06:30:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2024-09-14T06:30:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T06:30:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:30:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=a4067e44-69be-46b6-ae23-5ab5bfafc5ea 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror_parent 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local client_cluster=cluster1 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local remote_cluster=cluster1 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local uuid_var_name 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local peer_uuid 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T06:30:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:30:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-14T06:30:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ peer_uuid=c02c76e2-41a4-4781-85f9-7eac1962273f 2024-09-14T06:30:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T06:30:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:30:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T06:30:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 -eq 0 2024-09-14T06:30:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n octopus ']' 2024-09-14T06:30:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd set-require-min-compat-client octopus 2024-09-14T06:30:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:set require_min_compat_client to octopus 2024-09-14T06:30:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T06:30:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd set-require-min-compat-client octopus 2024-09-14T06:30:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:set require_min_compat_client to octopus 2024-09-14T06:30:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirrors cluster1 2024-09-14T06:30:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:30:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1:0 2024-09-14T06:30:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:30:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:30:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1:1 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:30:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:30:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-14T06:30:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirrors cluster2 2024-09-14T06:30:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:30:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster2:0 2024-09-14T06:30:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:0 2024-09-14T06:30:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:30:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster2:0 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster2 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster2 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:30:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster2:1 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:1 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster2:1 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:30:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster2 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster2 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:30:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-14T06:30:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-14T06:30:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.RQ7U70XNbJ/rbd-mirror.test.log 2024-09-14T06:30:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2024-09-14T06:30:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2024-09-14 06:30:21 TEST: add image and test replay after client crashes 2024-09-14T06:30:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14 06:30:21 TEST: add image and test replay after client crashes 2024-09-14T06:30:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=test 2024-09-14T06:30:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2024-09-14T06:30:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror test 512M 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:30:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:30:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:30:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:30:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2024-09-14T06:30:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:30:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:30:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:30:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:30:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:30:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:30:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:30:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:30:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:30:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:30:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:30:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:30:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:30:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:30:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:30:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:30:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:30:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:30:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:30:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:30:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T06:30:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:30:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:30:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:30:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:30:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:30:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:30:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:30:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:30:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:30:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:30:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:30:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:30:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:30:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T06:30:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T06:30:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:30:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:30:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:30:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 1 2024-09-14T06:30:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:30:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:30:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:30:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:30:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:30:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:30:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:30:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:30:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:30:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:30:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:30:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T06:30:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:30:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:30:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:30:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:30:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:30:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:30:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:30:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:30:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:30:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:30:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:30:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T06:30:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T06:30:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:30:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:30:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:30:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 2 2024-09-14T06:30:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:30:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:30:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:30:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:30:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:30:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:30:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:30:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:30:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:30:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:30:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:30:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:30:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:30:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:30:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name= 2024-09-14T06:30:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2024-09-14T06:30:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:30:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:30:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:30:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 72773632 bytes 2024-09-14T06:30:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 155453440 bytes 2024-09-14T06:30:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 244142080 bytes 2024-09-14T06:30:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 314486784 bytes 2024-09-14T06:30:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 332552192 bytes 2024-09-14T06:30:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 381023232 bytes 2024-09-14T06:30:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 435471360 bytes 2024-09-14T06:30:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 481067008 bytes 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:30:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:30:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:30:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:30:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:30:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:30:24 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:30:36' 2024-09-14T06:30:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:30:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:30:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:30:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-14T06:30:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:30:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:30:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:30:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:30:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:30:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:30:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:30:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:30:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:30:24' 2024-09-14T06:30:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:30:24' 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:30:24' 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:30:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:30:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42624267mirror.0smithi1702024-09-14 06:30:24cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:30:36' 2024-09-14T06:30:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:30:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:30:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:30:24' 2024-09-14T06:30:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:30:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:30:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:30:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:30:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:30:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap1 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2024-09-14T06:30:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:31:08.604 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:31:08.603+0000 7f566d49a640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:31:08.606 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:31:08.605+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-09-14T06:31:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:31:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:31:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:31:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:31:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:31:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:31:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:31:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:31:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:31:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:31:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:31:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:31:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:31:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:31:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:31:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:31:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:31:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:31:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:31:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:31:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:31:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:31:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:31:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:31:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 7 2024-09-14T06:31:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=27633], [object_number=28, tag_tid=2, entry_tid=27632], [object_number=31, tag_tid=2, entry_tid=27631], [object_number=30, tag_tid=2, entry_tid=27630]]], state=connected] 2024-09-14T06:31:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 7 2024-09-14T06:31:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=27633], [object_number=28, tag_tid=2, entry_tid=27632], [object_number=31, tag_tid=2, entry_tid=27631], [object_number=30, tag_tid=2, entry_tid=27630]]], state=connected] 2024-09-14T06:31:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=27633' 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=27633' 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=29, tag_tid=2, entry_tid=27633' '!=' '' 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=29, tag_tid=2, entry_tid=27633' = '' 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=29, tag_tid=2, entry_tid=27633' 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=27633' 2024-09-14T06:31:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=27633' 2024-09-14T06:31:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=27633 2024-09-14T06:31:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 27633 2024-09-14T06:31:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 7 2024-09-14T06:31:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=31, tag_tid=2, entry_tid=27903], [object_number=30, tag_tid=2, entry_tid=27902], [object_number=29, tag_tid=2, entry_tid=27901], [object_number=28, tag_tid=2, entry_tid=27900]]], state=connected] 2024-09-14T06:31:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 7 2024-09-14T06:31:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=31, tag_tid=2, entry_tid=27903], [object_number=30, tag_tid=2, entry_tid=27902], [object_number=29, tag_tid=2, entry_tid=27901], [object_number=28, tag_tid=2, entry_tid=27900]]], state=connected] 2024-09-14T06:31:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=31, tag_tid=2, entry_tid=27903' 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=31, tag_tid=2, entry_tid=27903' 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=31, tag_tid=2, entry_tid=27903' '!=' 'object_number=29, tag_tid=2, entry_tid=27633' 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=31, tag_tid=2, entry_tid=27903' = 'object_number=29, tag_tid=2, entry_tid=27633' 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=31, tag_tid=2, entry_tid=27903' 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=31, tag_tid=2, entry_tid=27903' 2024-09-14T06:31:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=31, tag_tid=2, entry_tid=27903' 2024-09-14T06:31:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=27903 2024-09-14T06:31:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 27903 2024-09-14T06:31:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 7 2024-09-14T06:31:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=28169], [object_number=28, tag_tid=2, entry_tid=28168], [object_number=31, tag_tid=2, entry_tid=28167], [object_number=30, tag_tid=2, entry_tid=28166]]], state=connected] 2024-09-14T06:31:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 7 2024-09-14T06:31:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=29, tag_tid=2, entry_tid=28169], [object_number=28, tag_tid=2, entry_tid=28168], [object_number=31, tag_tid=2, entry_tid=28167], [object_number=30, tag_tid=2, entry_tid=28166]]], state=connected] 2024-09-14T06:31:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=29, tag_tid=2, entry_tid=28169' 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=29, tag_tid=2, entry_tid=28169' 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=29, tag_tid=2, entry_tid=28169' '!=' 'object_number=31, tag_tid=2, entry_tid=27903' 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=29, tag_tid=2, entry_tid=28169' = 'object_number=31, tag_tid=2, entry_tid=27903' 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=29, tag_tid=2, entry_tid=28169' 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=28169' 2024-09-14T06:31:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=29, tag_tid=2, entry_tid=28169' 2024-09-14T06:31:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=28169 2024-09-14T06:31:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 28169 2024-09-14T06:31:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=28274], [object_number=33, tag_tid=2, entry_tid=28273], [object_number=32, tag_tid=2, entry_tid=28272], [object_number=35, tag_tid=2, entry_tid=28271]]], state=connected] 2024-09-14T06:31:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=28274], [object_number=33, tag_tid=2, entry_tid=28273], [object_number=32, tag_tid=2, entry_tid=28272], [object_number=35, tag_tid=2, entry_tid=28271]]], state=connected] 2024-09-14T06:31:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=34, tag_tid=2, entry_tid=28274' 2024-09-14T06:31:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=34, tag_tid=2, entry_tid=28274' 2024-09-14T06:31:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=28274' '!=' 'object_number=29, tag_tid=2, entry_tid=28169' 2024-09-14T06:31:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=28274' = 'object_number=29, tag_tid=2, entry_tid=28169' 2024-09-14T06:31:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=34, tag_tid=2, entry_tid=28274' 2024-09-14T06:31:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=28274' 2024-09-14T06:31:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=28274' 2024-09-14T06:31:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=28274 2024-09-14T06:31:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 28274 2024-09-14T06:31:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=28545], [object_number=32, tag_tid=2, entry_tid=28544], [object_number=35, tag_tid=2, entry_tid=28543], [object_number=34, tag_tid=2, entry_tid=28542]]], state=connected] 2024-09-14T06:31:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=28545], [object_number=32, tag_tid=2, entry_tid=28544], [object_number=35, tag_tid=2, entry_tid=28543], [object_number=34, tag_tid=2, entry_tid=28542]]], state=connected] 2024-09-14T06:31:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=28545' 2024-09-14T06:31:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=28545' 2024-09-14T06:31:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=28545' '!=' 'object_number=34, tag_tid=2, entry_tid=28274' 2024-09-14T06:31:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=28545' = 'object_number=34, tag_tid=2, entry_tid=28274' 2024-09-14T06:31:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=33, tag_tid=2, entry_tid=28545' 2024-09-14T06:31:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=28545' 2024-09-14T06:31:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=28545' 2024-09-14T06:31:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=28545 2024-09-14T06:31:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 28545 2024-09-14T06:31:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=28786], [object_number=33, tag_tid=2, entry_tid=28785], [object_number=32, tag_tid=2, entry_tid=28784], [object_number=35, tag_tid=2, entry_tid=28783]]], state=connected] 2024-09-14T06:31:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=28786], [object_number=33, tag_tid=2, entry_tid=28785], [object_number=32, tag_tid=2, entry_tid=28784], [object_number=35, tag_tid=2, entry_tid=28783]]], state=connected] 2024-09-14T06:31:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=34, tag_tid=2, entry_tid=28786' 2024-09-14T06:31:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=34, tag_tid=2, entry_tid=28786' 2024-09-14T06:31:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=28786' '!=' 'object_number=33, tag_tid=2, entry_tid=28545' 2024-09-14T06:31:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=28786' = 'object_number=33, tag_tid=2, entry_tid=28545' 2024-09-14T06:31:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=34, tag_tid=2, entry_tid=28786' 2024-09-14T06:31:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=28786' 2024-09-14T06:31:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=28786' 2024-09-14T06:31:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=28786 2024-09-14T06:31:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 28786 2024-09-14T06:31:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29032], [object_number=35, tag_tid=2, entry_tid=29031], [object_number=34, tag_tid=2, entry_tid=29030], [object_number=33, tag_tid=2, entry_tid=29029]]], state=connected] 2024-09-14T06:31:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29032], [object_number=35, tag_tid=2, entry_tid=29031], [object_number=34, tag_tid=2, entry_tid=29030], [object_number=33, tag_tid=2, entry_tid=29029]]], state=connected] 2024-09-14T06:31:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=32, tag_tid=2, entry_tid=29032' 2024-09-14T06:31:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=32, tag_tid=2, entry_tid=29032' 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29032' '!=' 'object_number=34, tag_tid=2, entry_tid=28786' 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29032' = 'object_number=34, tag_tid=2, entry_tid=28786' 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=32, tag_tid=2, entry_tid=29032' 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29032' 2024-09-14T06:31:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29032' 2024-09-14T06:31:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:14.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=29032 2024-09-14T06:31:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 29032 2024-09-14T06:31:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29284], [object_number=35, tag_tid=2, entry_tid=29283], [object_number=34, tag_tid=2, entry_tid=29282], [object_number=33, tag_tid=2, entry_tid=29281]]], state=connected] 2024-09-14T06:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29284], [object_number=35, tag_tid=2, entry_tid=29283], [object_number=34, tag_tid=2, entry_tid=29282], [object_number=33, tag_tid=2, entry_tid=29281]]], state=connected] 2024-09-14T06:31:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=32, tag_tid=2, entry_tid=29284' 2024-09-14T06:31:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=32, tag_tid=2, entry_tid=29284' 2024-09-14T06:31:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29284' '!=' 'object_number=32, tag_tid=2, entry_tid=29032' 2024-09-14T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29284' = 'object_number=32, tag_tid=2, entry_tid=29032' 2024-09-14T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=32, tag_tid=2, entry_tid=29284' 2024-09-14T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29284' 2024-09-14T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29284' 2024-09-14T06:31:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=29284 2024-09-14T06:31:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 29284 2024-09-14T06:31:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29544], [object_number=35, tag_tid=2, entry_tid=29543], [object_number=34, tag_tid=2, entry_tid=29542], [object_number=33, tag_tid=2, entry_tid=29541]]], state=connected] 2024-09-14T06:31:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29544], [object_number=35, tag_tid=2, entry_tid=29543], [object_number=34, tag_tid=2, entry_tid=29542], [object_number=33, tag_tid=2, entry_tid=29541]]], state=connected] 2024-09-14T06:31:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=32, tag_tid=2, entry_tid=29544' 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=32, tag_tid=2, entry_tid=29544' 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29544' '!=' 'object_number=32, tag_tid=2, entry_tid=29284' 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29544' = 'object_number=32, tag_tid=2, entry_tid=29284' 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=32, tag_tid=2, entry_tid=29544' 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29544' 2024-09-14T06:31:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29544' 2024-09-14T06:31:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=29544 2024-09-14T06:31:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 29544 2024-09-14T06:31:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29920], [object_number=35, tag_tid=2, entry_tid=29919], [object_number=34, tag_tid=2, entry_tid=29918], [object_number=33, tag_tid=2, entry_tid=29917]]], state=connected] 2024-09-14T06:31:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=29920], [object_number=35, tag_tid=2, entry_tid=29919], [object_number=34, tag_tid=2, entry_tid=29918], [object_number=33, tag_tid=2, entry_tid=29917]]], state=connected] 2024-09-14T06:31:16.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=32, tag_tid=2, entry_tid=29920' 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=32, tag_tid=2, entry_tid=29920' 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29920' '!=' 'object_number=32, tag_tid=2, entry_tid=29544' 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=29920' = 'object_number=32, tag_tid=2, entry_tid=29544' 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=32, tag_tid=2, entry_tid=29920' 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29920' 2024-09-14T06:31:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=29920' 2024-09-14T06:31:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=29920 2024-09-14T06:31:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 29920 2024-09-14T06:31:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=30188], [object_number=35, tag_tid=2, entry_tid=30187], [object_number=34, tag_tid=2, entry_tid=30186], [object_number=33, tag_tid=2, entry_tid=30185]]], state=connected] 2024-09-14T06:31:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=32, tag_tid=2, entry_tid=30188], [object_number=35, tag_tid=2, entry_tid=30187], [object_number=34, tag_tid=2, entry_tid=30186], [object_number=33, tag_tid=2, entry_tid=30185]]], state=connected] 2024-09-14T06:31:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=32, tag_tid=2, entry_tid=30188' 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=32, tag_tid=2, entry_tid=30188' 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=30188' '!=' 'object_number=32, tag_tid=2, entry_tid=29920' 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=32, tag_tid=2, entry_tid=30188' = 'object_number=32, tag_tid=2, entry_tid=29920' 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=32, tag_tid=2, entry_tid=30188' 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=30188' 2024-09-14T06:31:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=32, tag_tid=2, entry_tid=30188' 2024-09-14T06:31:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=30188 2024-09-14T06:31:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 30188 2024-09-14T06:31:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=30454], [object_number=33, tag_tid=2, entry_tid=30453], [object_number=32, tag_tid=2, entry_tid=30452], [object_number=35, tag_tid=2, entry_tid=30451]]], state=connected] 2024-09-14T06:31:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=30454], [object_number=33, tag_tid=2, entry_tid=30453], [object_number=32, tag_tid=2, entry_tid=30452], [object_number=35, tag_tid=2, entry_tid=30451]]], state=connected] 2024-09-14T06:31:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=34, tag_tid=2, entry_tid=30454' 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=34, tag_tid=2, entry_tid=30454' 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=30454' '!=' 'object_number=32, tag_tid=2, entry_tid=30188' 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=30454' = 'object_number=32, tag_tid=2, entry_tid=30188' 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=34, tag_tid=2, entry_tid=30454' 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=30454' 2024-09-14T06:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=30454' 2024-09-14T06:31:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=30454 2024-09-14T06:31:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 30454 2024-09-14T06:31:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=30698], [object_number=33, tag_tid=2, entry_tid=30697], [object_number=32, tag_tid=2, entry_tid=30696], [object_number=35, tag_tid=2, entry_tid=30695]]], state=connected] 2024-09-14T06:31:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=34, tag_tid=2, entry_tid=30698], [object_number=33, tag_tid=2, entry_tid=30697], [object_number=32, tag_tid=2, entry_tid=30696], [object_number=35, tag_tid=2, entry_tid=30695]]], state=connected] 2024-09-14T06:31:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=34, tag_tid=2, entry_tid=30698' 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=34, tag_tid=2, entry_tid=30698' 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=30698' '!=' 'object_number=34, tag_tid=2, entry_tid=30454' 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=34, tag_tid=2, entry_tid=30698' = 'object_number=34, tag_tid=2, entry_tid=30454' 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=34, tag_tid=2, entry_tid=30698' 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=30698' 2024-09-14T06:31:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=34, tag_tid=2, entry_tid=30698' 2024-09-14T06:31:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=30698 2024-09-14T06:31:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 30698 2024-09-14T06:31:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=30945], [object_number=32, tag_tid=2, entry_tid=30944], [object_number=35, tag_tid=2, entry_tid=30943], [object_number=34, tag_tid=2, entry_tid=30942]]], state=connected] 2024-09-14T06:31:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=33, tag_tid=2, entry_tid=30945], [object_number=32, tag_tid=2, entry_tid=30944], [object_number=35, tag_tid=2, entry_tid=30943], [object_number=34, tag_tid=2, entry_tid=30942]]], state=connected] 2024-09-14T06:31:17.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=33, tag_tid=2, entry_tid=30945' 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=33, tag_tid=2, entry_tid=30945' 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=30945' '!=' 'object_number=34, tag_tid=2, entry_tid=30698' 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=33, tag_tid=2, entry_tid=30945' = 'object_number=34, tag_tid=2, entry_tid=30698' 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=33, tag_tid=2, entry_tid=30945' 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=30945' 2024-09-14T06:31:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=2, entry_tid=30945' 2024-09-14T06:31:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=30945 2024-09-14T06:31:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 30945 2024-09-14T06:31:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174], [object_number=33, tag_tid=2, entry_tid=31173], [object_number=32, tag_tid=2, entry_tid=31172]]], state=connected] 2024-09-14T06:31:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174], [object_number=33, tag_tid=2, entry_tid=31173], [object_number=32, tag_tid=2, entry_tid=31172]]], state=connected] 2024-09-14T06:31:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=35, tag_tid=2, entry_tid=31175' 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=35, tag_tid=2, entry_tid=31175' 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=35, tag_tid=2, entry_tid=31175' '!=' 'object_number=33, tag_tid=2, entry_tid=30945' 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=35, tag_tid=2, entry_tid=31175' = 'object_number=33, tag_tid=2, entry_tid=30945' 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=35, tag_tid=2, entry_tid=31175' 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:31:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=35, tag_tid=2, entry_tid=31175' 2024-09-14T06:31:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:31:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:31:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:31:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=35, tag_tid=2, entry_tid=31175' 2024-09-14T06:31:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:31:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:31:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=31175 2024-09-14T06:31:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 31175 2024-09-14T06:31:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:31:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:31:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:31:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:31:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:31:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:31:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:31:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:31:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:31:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:31:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:31:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:31:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:31:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:31:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:31:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:31:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:31:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:31:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:31:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:31:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:31:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:31:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:31:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:31:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:31:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 8 2024-09-14T06:31:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 8 2024-09-14T06:31:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:31:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=33, tag_tid=3, entry_tid=1], [object_number=32, tag_tid=3, entry_tid=0], [object_number=35, tag_tid=2, entry_tid=31175], [object_number=34, tag_tid=2, entry_tid=31174]]], state=connected] 2024-09-14T06:31:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:31:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=33, tag_tid=3, entry_tid=1' -a 'object_number=33, tag_tid=3, entry_tid=1' = 'object_number=33, tag_tid=3, entry_tid=1' 2024-09-14T06:31:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-14T06:31:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2024-09-14T06:31:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:31:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:31:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:31:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-14T06:31:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:31:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:31:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:31:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:31:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:31:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:31:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:31:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:31:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:31:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:31:20.377 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:31:20.374+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:31:20.485 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:31:20.483+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:31:20.486 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:31:20.483+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:31:20.486 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:31:20.483+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:31:20.700 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:31:20.698+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:31:20.703 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:31:20.701+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:31:20.704 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:31:20.702+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:31:20.756 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:31:20.754+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:31:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:31:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:31:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:31:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap1-clean 2024-09-14T06:31:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-14T06:31:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:31:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1-clean 2024-09-14T06:31:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:31:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:31:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:31:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:31:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:31:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2024-09-14T06:31:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:31:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:31:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 39136256 bytes 2024-09-14T06:31:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 57270272 bytes 2024-09-14T06:31:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 57881600 bytes 2024-09-14T06:31:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 91030528 bytes 2024-09-14T06:31:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 136584192 bytes 2024-09-14T06:31:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 160148480 bytes 2024-09-14T06:31:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 240310272 bytes 2024-09-14T06:31:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 289968128 bytes 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:31:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:31:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:31:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:31:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:31:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:31:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:31:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:31:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:31:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":9448903.6,"entries_behind_primary":0,"entries_per_second":588.9299999999999,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":3}} 2024-09-14T06:31:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:31:21 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:31:31' 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:31:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:31:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":9448903.6,"entries_behind_primary":0,"entries_per_second":588.9299999999999,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":3}} 2024-09-14T06:31:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:31:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:31:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:31:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:31:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:31:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:31:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:31:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:31:21' 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:31:21' 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:31:21' 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:31:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":9448903.6,"entries_behind_primary":0,"entries_per_second":588.9299999999999,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":3},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":3}}42624267mirror.0smithi1702024-09-14 06:31:21cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:31:31' 2024-09-14T06:31:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:31:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:31:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:31:21' 2024-09-14T06:31:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:31:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:31:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:31:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:31:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:31:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2 2024-09-14T06:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap2 2024-09-14T06:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2024-09-14T06:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:31:59.651 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:31:59.650+0000 7ff0846f3640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-09-14T06:31:59.654 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:31:59.653+0000 7ff07d6e5640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-09-14T06:32:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:32:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:32:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:32:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:32:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:32:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:32:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:32:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:32:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:32:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:32:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:32:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:32:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:32:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:32:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:32:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:32:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:32:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:32:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:32:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:32:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:32:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:32:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:32:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:32:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:32:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:32:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:32:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:32:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:32:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:32:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:32:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:32:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:32:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:32:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:32:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:32:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:32:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:32:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:32:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:32:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:32:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:32:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:32:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:32:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 14 2024-09-14T06:32:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 14 2024-09-14T06:32:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:32:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=6, entry_tid=1], [object_number=56, tag_tid=6, entry_tid=0], [object_number=59, tag_tid=5, entry_tid=19887], [object_number=58, tag_tid=5, entry_tid=19886]]], state=connected] 2024-09-14T06:32:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=57, tag_tid=6, entry_tid=1], [object_number=56, tag_tid=6, entry_tid=0], [object_number=59, tag_tid=5, entry_tid=19887], [object_number=58, tag_tid=5, entry_tid=19886]]], state=connected] 2024-09-14T06:32:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=57, tag_tid=6, entry_tid=1' 2024-09-14T06:32:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:32:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:32:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:32:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:32:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:32:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:32:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 14 2024-09-14T06:32:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 14 2024-09-14T06:32:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:32:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=57, tag_tid=6, entry_tid=1], [object_number=56, tag_tid=6, entry_tid=0], [object_number=59, tag_tid=5, entry_tid=19887], [object_number=58, tag_tid=5, entry_tid=19886]]], state=connected] 2024-09-14T06:32:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=57, tag_tid=6, entry_tid=1], [object_number=56, tag_tid=6, entry_tid=0], [object_number=59, tag_tid=5, entry_tid=19887], [object_number=58, tag_tid=5, entry_tid=19886]]], state=connected] 2024-09-14T06:32:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=57, tag_tid=6, entry_tid=1' 2024-09-14T06:32:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=57, tag_tid=6, entry_tid=1' -a 'object_number=57, tag_tid=6, entry_tid=1' = 'object_number=57, tag_tid=6, entry_tid=1' 2024-09-14T06:32:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-14T06:32:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2024-09-14T06:32:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:32:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:32:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-14T06:32:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:32:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:32:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:32:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 3 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:31:24 2024 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:32:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:32:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:32:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:32:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap1-clean ']' 2024-09-14T06:32:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-14T06:32:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1-clean 2024-09-14T06:32:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:32:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:05.961 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:05.960+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:32:05.965 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:05.964+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:32:06.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:06.282+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:32:06.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:06.282+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:32:06.287 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:06.286+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:06.287 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:06.286+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:06.289 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:06.287+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:06.341 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:06.340+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:32:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:07.389 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.388+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:32:07.390 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.389+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:32:07.390 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.389+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:32:07.494 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.493+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:32:07.496 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.495+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:32:07.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:07.558+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:07.562 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:07.560+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:07.568 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:07.566+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:07.664 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.663+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:32:07.665 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.663+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:32:07.665 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.663+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:32:07.735 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.734+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:32:07.736 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:07.735+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:32:07.737 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:07.736+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:32:07.804 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:07.803+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:32:07.805 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:07.804+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:32:07.932 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:07.931+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:07.934 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:07.933+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:07.936 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:07.935+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:07.997 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.996+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:32:07.998 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:07.997+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:32:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap2-clean 2024-09-14T06:32:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-14T06:32:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2-clean 2024-09-14T06:32:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:32:08.673 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:32:08.672+0000 7f5669c93640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:32:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:32:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:32:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:32:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2024-09-14T06:32:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:32:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:32:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 24663040 bytes 2024-09-14T06:32:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 66461696 bytes 2024-09-14T06:32:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 134940672 bytes 2024-09-14T06:32:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:32:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:32:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:32:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:32:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:32:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:32:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:32:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:32:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:32:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:32:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:32:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:32:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:32:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:32:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:32:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":6758225.07,"entries_behind_primary":0,"entries_per_second":427.73,"non_primary_position":{"entry_tid":8499,"object_number":47,"tag_tid":5},"primary_position":{"entry_tid":7623,"object_number":47,"tag_tid":5}} 2024-09-14T06:32:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:31:51 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:32:11' 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:32:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:32:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":6758225.07,"entries_behind_primary":0,"entries_per_second":427.73,"non_primary_position":{"entry_tid":8499,"object_number":47,"tag_tid":5},"primary_position":{"entry_tid":7623,"object_number":47,"tag_tid":5}} 2024-09-14T06:32:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:32:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:32:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:32:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:32:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:32:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:32:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:32:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:31:51' 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:31:51' 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:31:51' 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:32:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":6758225.07,"entries_behind_primary":0,"entries_per_second":427.73,"non_primary_position":{"entry_tid":8499,"object_number":47,"tag_tid":5},"primary_position":{"entry_tid":7623,"object_number":47,"tag_tid":5}}42624267mirror.0smithi1702024-09-14 06:31:51cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:32:11' 2024-09-14T06:32:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:32:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:32:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:31:51' 2024-09-14T06:32:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:32:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:32:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:32:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:32:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:32:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3 2024-09-14T06:32:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap3 2024-09-14T06:32:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2024-09-14T06:32:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:32:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:32:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:32:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:32:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:32:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:32:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:32:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:32:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:32:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:32:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:32:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:32:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:32:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:32:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:32:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:32:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:32:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:32:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:32:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:32:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:32:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:32:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:32:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:32:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:32:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:32:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 16 2024-09-14T06:32:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 17 2024-09-14T06:32:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:32:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=9, entry_tid=1], [object_number=68, tag_tid=9, entry_tid=0], [object_number=67, tag_tid=8, entry_tid=7467], [object_number=66, tag_tid=8, entry_tid=7466]]], state=connected] 2024-09-14T06:32:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=69, tag_tid=9, entry_tid=1], [object_number=68, tag_tid=9, entry_tid=0], [object_number=67, tag_tid=8, entry_tid=7467], [object_number=66, tag_tid=8, entry_tid=7466]]], state=connected] 2024-09-14T06:32:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=69, tag_tid=9, entry_tid=1' 2024-09-14T06:32:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:32:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:32:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 16 2024-09-14T06:32:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 17 2024-09-14T06:32:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:32:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=69, tag_tid=9, entry_tid=1], [object_number=68, tag_tid=9, entry_tid=0], [object_number=67, tag_tid=8, entry_tid=7467], [object_number=66, tag_tid=8, entry_tid=7466]]], state=connected] 2024-09-14T06:32:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=69, tag_tid=9, entry_tid=1], [object_number=68, tag_tid=9, entry_tid=0], [object_number=67, tag_tid=8, entry_tid=7467], [object_number=66, tag_tid=8, entry_tid=7466]]], state=connected] 2024-09-14T06:32:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=69, tag_tid=9, entry_tid=1' 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=69, tag_tid=9, entry_tid=1' -a 'object_number=69, tag_tid=9, entry_tid=1' = 'object_number=69, tag_tid=9, entry_tid=1' 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:32:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:32:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:32:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-14T06:32:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:32:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:32:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:32:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 5 2024-09-14T06:32:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:32:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:32:05 2024 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:32:25 2024 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:32:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap2-clean ']' 2024-09-14T06:32:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2-clean 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:51.056 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:51.055+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:51.060 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:51.058+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:51.064 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:51.062+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:51.174 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:51.173+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:32:51.175 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:51.174+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:32:51.175 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:51.174+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:32:51.179 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:51.178+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:32:51.349 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:51.347+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:32:51.353 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:51.352+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:32:51.359 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:51.357+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:32:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:32:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:52.659 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:52.657+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:32:52.660 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:52.658+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:52.660 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:52.659+0000 7f14d881b640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:52.660 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:52.659+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:52.772 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:52.771+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:32:52.774 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:52.773+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:32:52.776 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:52.775+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:32:52.824 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:52.823+0000 7f641dd1b640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:32:52.952 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:52.951+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:32:52.955 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:32:52.954+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:32:52.962 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:52.961+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:52.963 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:52.962+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:32:53.070 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:53.068+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:32:53.071 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:53.070+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:32:53.071 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:53.070+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:32:53.133 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:32:53.132+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:32:53.135 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:53.134+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:32:53.140 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:53.139+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:32:53.144 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:32:53.143+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:32:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:32:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap3-clean 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3-clean 2024-09-14T06:32:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:32:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:32:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:32:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:32:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:32:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:32:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2024-09-14T06:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:32:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 39609344 bytes 2024-09-14T06:32:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 85575680 bytes 2024-09-14T06:32:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 133888000 bytes 2024-09-14T06:33:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:33:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:33:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:33:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:33:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:33:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:33:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:33:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":2270073.83,"entries_behind_primary":0,"entries_per_second":142.2,"non_primary_position":{"entry_tid":1,"object_number":69,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":9}} 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:32:51 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:33:01' 2024-09-14T06:33:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:33:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:33:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:33:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":2270073.83,"entries_behind_primary":0,"entries_per_second":142.2,"non_primary_position":{"entry_tid":1,"object_number":69,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":9}} 2024-09-14T06:33:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:33:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:33:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:33:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:33:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:33:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:33:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:33:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:32:51' 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:32:51' 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:32:51' 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:33:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:33:03.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":2270073.83,"entries_behind_primary":0,"entries_per_second":142.2,"non_primary_position":{"entry_tid":1,"object_number":69,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":69,"tag_tid":9}}42624267mirror.0smithi1702024-09-14 06:32:51cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:33:01' 2024-09-14T06:33:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:33:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:33:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:32:51' 2024-09-14T06:33:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:33:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:33:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:33:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:33:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap4 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2024-09-14T06:33:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:33:31.748 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:33:31.746+0000 7ff07e6e7640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-09-14T06:33:31.750 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:33:31.749+0000 7ff0846f3640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-09-14T06:33:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:33:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:33:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:33:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:33:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:33:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:33:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:33:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:33:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:33:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:33:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:33:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:33:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:33:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:33:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:33:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:33:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:33:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:33:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:33:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:33:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:33:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:33:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:33:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:33:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:33:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:33:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:33:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:33:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:33:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:33:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:33:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:33:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:33:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:33:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:33:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:33:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:33:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:33:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:33:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:33:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:33:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:33:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:33:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:33:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:33:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:33:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:33:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:33:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:33:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:33:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:33:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:33:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:33:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:33:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:33:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:33:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:33:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:33:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:33:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:33:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:33:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:33:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:33:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:33:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:33:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:33:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:33:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:33:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:33:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:33:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:33:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:33:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:33:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:33:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:33:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:33:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:33:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:33:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:33:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:33:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 20 2024-09-14T06:33:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 20 2024-09-14T06:33:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:33:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=81, tag_tid=12, entry_tid=1], [object_number=80, tag_tid=12, entry_tid=0], [object_number=82, tag_tid=11, entry_tid=13134], [object_number=83, tag_tid=11, entry_tid=13131]]], state=connected] 2024-09-14T06:33:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=81, tag_tid=12, entry_tid=1], [object_number=80, tag_tid=12, entry_tid=0], [object_number=82, tag_tid=11, entry_tid=13134], [object_number=83, tag_tid=11, entry_tid=13131]]], state=connected] 2024-09-14T06:33:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:33:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=81, tag_tid=12, entry_tid=1' 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:33:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:33:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:33:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:33:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:33:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:33:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:33:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:33:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:33:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 20 2024-09-14T06:33:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 20 2024-09-14T06:33:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:33:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=81, tag_tid=12, entry_tid=1], [object_number=80, tag_tid=12, entry_tid=0], [object_number=82, tag_tid=11, entry_tid=13134], [object_number=83, tag_tid=11, entry_tid=13131]]], state=connected] 2024-09-14T06:33:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=81, tag_tid=12, entry_tid=1], [object_number=80, tag_tid=12, entry_tid=0], [object_number=82, tag_tid=11, entry_tid=13134], [object_number=83, tag_tid=11, entry_tid=13131]]], state=connected] 2024-09-14T06:33:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=81, tag_tid=12, entry_tid=1' 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=81, tag_tid=12, entry_tid=1' -a 'object_number=81, tag_tid=12, entry_tid=1' = 'object_number=81, tag_tid=12, entry_tid=1' 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:33:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:33:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:33:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 7 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:33:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:32:05 2024 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:32:25 2024 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:33:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:33:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap3-clean ']' 2024-09-14T06:33:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3-clean 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:33:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:33:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:37.097 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:37.095+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:37.100 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:37.098+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:33:37.208 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:37.207+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:33:37.211 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:37.210+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:33:37.212 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:37.211+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:33:37.214 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:37.212+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:33:37.266 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:33:37.265+0000 7f6419d13640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:33:37.269 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:33:37.267+0000 7f641f51e640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:33:37.401 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:37.400+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:33:37.401 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:37.400+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:33:37.404 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:37.403+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:37.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:37.463+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:37.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:37.464+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:33:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:33:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:33:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:38.556 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:38.554+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:33:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:38.701+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:38.705 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:38.704+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:38.705 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:38.704+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:38.767 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:38.766+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:33:38.769 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:38.768+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:33:38.772 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:38.770+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:33:38.772 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:38.771+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:38.825 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:38.824+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:33:38.827 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:38.826+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:33:38.832 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:38.831+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:33:38.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:33:38.832+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:33:38.896 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:33:38.895+0000 7f6419d13640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:33:38.897 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:33:38.896+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:33:39.027 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:39.026+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:33:39.029 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:39.028+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:33:39.033 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:33:39.031+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:33:39.033 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:33:39.031+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:33:39.102 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:39.100+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:33:39.102 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:33:39.101+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:33:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:33:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:33:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap4-clean 2024-09-14T06:33:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-14T06:33:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:33:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4-clean 2024-09-14T06:33:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:33:39.669 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:33:39.668+0000 7f0c59c0b640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2024-09-14T06:33:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:33:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:33:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:33:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:33:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:33:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:33:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:33:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 29933568 bytes 2024-09-14T06:33:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 47091712 bytes 2024-09-14T06:33:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 63147008 bytes 2024-09-14T06:33:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 110779392 bytes 2024-09-14T06:33:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 127306752 bytes 2024-09-14T06:33:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 170890240 bytes 2024-09-14T06:33:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 193806336 bytes 2024-09-14T06:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 200101888 bytes 2024-09-14T06:33:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:900 IOs, 234414080 bytes 2024-09-14T06:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:33:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:33:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:33:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:33:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:33:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:33:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":2864414.93,"entries_behind_primary":1,"entries_per_second":177.17,"non_primary_position":{"entry_tid":1,"object_number":81,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":81,"tag_tid":13},"seconds_until_synced":0} 2024-09-14T06:33:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:33:51 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:33:51' 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:33:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:33:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:33:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":2864414.93,"entries_behind_primary":1,"entries_per_second":177.17,"non_primary_position":{"entry_tid":1,"object_number":81,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":81,"tag_tid":13},"seconds_until_synced":0} 2024-09-14T06:33:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:33:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:33:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:33:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:33:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:33:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:33:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:33:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:33:51' 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:33:51' 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:33:51' 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:33:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:33:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":2864414.93,"entries_behind_primary":1,"entries_per_second":177.17,"non_primary_position":{"entry_tid":1,"object_number":81,"tag_tid":12},"primary_position":{"entry_tid":1,"object_number":81,"tag_tid":13},"seconds_until_synced":0}42624267mirror.0smithi1702024-09-14 06:33:51cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:33:51' 2024-09-14T06:33:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:33:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:33:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:33:51' 2024-09-14T06:33:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:33:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:33:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:33:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:33:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap5 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:33:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2024-09-14T06:33:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:34:21.801 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:34:21.800+0000 7f0c5a40c640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-09-14T06:34:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:34:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:34:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:34:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:34:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:34:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:34:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:34:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:34:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:34:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:34:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:34:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:34:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:34:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:34:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:34:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:34:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:34:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:34:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:34:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:34:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:34:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:34:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:34:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:34:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:34:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:34:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:34:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:34:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:34:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:34:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:34:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:34:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:34:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:34:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:34:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:34:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:34:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:34:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:34:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:34:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:34:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:34:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:34:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:34:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:34:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:34:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:34:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:34:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:34:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:34:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:34:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:34:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:34:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:34:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:34:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 25 2024-09-14T06:34:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 26 2024-09-14T06:34:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:34:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=15, entry_tid=1], [object_number=104, tag_tid=15, entry_tid=0], [object_number=103, tag_tid=14, entry_tid=17327], [object_number=102, tag_tid=14, entry_tid=17326]]], state=connected] 2024-09-14T06:34:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=105, tag_tid=15, entry_tid=1], [object_number=104, tag_tid=15, entry_tid=0], [object_number=103, tag_tid=14, entry_tid=17327], [object_number=102, tag_tid=14, entry_tid=17326]]], state=connected] 2024-09-14T06:34:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:34:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=105, tag_tid=15, entry_tid=1' 2024-09-14T06:34:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:34:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:34:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:34:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:34:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:34:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 25 2024-09-14T06:34:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 26 2024-09-14T06:34:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:34:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=105, tag_tid=15, entry_tid=1], [object_number=104, tag_tid=15, entry_tid=0], [object_number=103, tag_tid=14, entry_tid=17327], [object_number=102, tag_tid=14, entry_tid=17326]]], state=connected] 2024-09-14T06:34:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=105, tag_tid=15, entry_tid=1], [object_number=104, tag_tid=15, entry_tid=0], [object_number=103, tag_tid=14, entry_tid=17327], [object_number=102, tag_tid=14, entry_tid=17326]]], state=connected] 2024-09-14T06:34:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:34:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=105, tag_tid=15, entry_tid=1' 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=105, tag_tid=15, entry_tid=1' -a 'object_number=105, tag_tid=15, entry_tid=1' = 'object_number=105, tag_tid=15, entry_tid=1' 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:34:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:34:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-14T06:34:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 9 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:33:36 2024 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:33:42 2024 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:34:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:34:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:34:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:34:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:34:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:34:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap4-clean ']' 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4-clean 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:34:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:34:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:26.207 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.206+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:34:26.210 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.208+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:26.210 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.208+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:26.211 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.210+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:26.294 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:26.293+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:34:26.298 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:26.297+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:34:26.298 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:26.297+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:34:26.430 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.429+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:34:26.491 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.490+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:34:26.491 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.490+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:34:26.496 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.495+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:26.554 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.553+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:34:26.554 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.553+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:34:26.554 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.553+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:34:26.555 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.554+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:34:26.560 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.558+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:26.561 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:26.560+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:26.622 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:26.621+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:34:26.625 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:26.624+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:34:26.631 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:26.629+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:34:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:34:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5 2024-09-14T06:34:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2024-09-14T06:34:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:34:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:34:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:34:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:34:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:27.685 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:27.682+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:34:27.685 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:27.683+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:34:27.685 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:27.684+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:34:27.685 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:27.684+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:34:27.863 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:27.862+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:27.863 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:27.862+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:34:27.864 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:27.863+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:27.939 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:27.938+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:34:27.939 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:27.938+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:34:28.000 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:27.998+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:34:28.004 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:28.003+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:34:28.005 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:28.004+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:34:28.070 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:28.069+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:34:28.070 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:28.069+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:34:28.079 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:34:28.078+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:34:28.140 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:34:28.139+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:34:28.141 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:28.140+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:34:28.143 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:34:28.142+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:34:28.144 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:28.143+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:34:28.147 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:34:28.146+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:34:28.273 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:28.272+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:34:28.274 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:28.273+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:34:28.275 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:28.274+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:34:28.278 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:34:28.277+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:34:28.345 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:28.344+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:34:28.346 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:34:28.345+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:34:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:34:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:34:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap5-clean 2024-09-14T06:34:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-14T06:34:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:34:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5-clean 2024-09-14T06:34:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:34:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2024-09-14T06:34:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:34:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:34:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 67094528 bytes 2024-09-14T06:34:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 113696768 bytes 2024-09-14T06:34:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 122889216 bytes 2024-09-14T06:34:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 123817984 bytes 2024-09-14T06:34:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 128558080 bytes 2024-09-14T06:34:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 129173504 bytes 2024-09-14T06:34:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 191106048 bytes 2024-09-14T06:34:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:34:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:34:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:34:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:34:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:34:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:34:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:34:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":7671347.87,"entries_behind_primary":0,"entries_per_second":486.77,"non_primary_position":{"entry_tid":15416,"object_number":100,"tag_tid":14},"primary_position":{"entry_tid":14931,"object_number":103,"tag_tid":14}} 2024-09-14T06:34:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:34:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:34:21 2024-09-14T06:34:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:34:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:34:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:34:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:34:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:34:36' 2024-09-14T06:34:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:34:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:34:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:34:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":7671347.87,"entries_behind_primary":0,"entries_per_second":486.77,"non_primary_position":{"entry_tid":15416,"object_number":100,"tag_tid":14},"primary_position":{"entry_tid":14931,"object_number":103,"tag_tid":14}} 2024-09-14T06:34:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:34:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:34:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:34:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:34:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:34:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:34:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:34:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:34:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:34:21' 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:34:21' 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:34:21' 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:34:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:34:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":7671347.87,"entries_behind_primary":0,"entries_per_second":486.77,"non_primary_position":{"entry_tid":15416,"object_number":100,"tag_tid":14},"primary_position":{"entry_tid":14931,"object_number":103,"tag_tid":14}}42624267mirror.0smithi1702024-09-14 06:34:21cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:34:36' 2024-09-14T06:34:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:34:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:34:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:34:21' 2024-09-14T06:34:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:34:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:34:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:34:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:34:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:34:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:34:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6 2024-09-14T06:34:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap6 2024-09-14T06:34:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:34:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:34:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:34:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2024-09-14T06:34:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:35:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:35:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:35:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:35:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:35:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:35:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:35:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:35:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:35:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:35:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:35:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:35:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:35:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:35:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:35:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:35:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:35:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:35:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:35:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:35:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:35:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:35:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:35:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:35:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:35:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:35:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:35:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:35:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:35:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:35:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:35:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:35:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:35:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:35:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:35:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:35:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:35:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:35:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:35:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:35:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:35:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:35:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:35:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:35:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:35:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:35:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:35:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:35:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:35:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:35:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:35:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:35:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:35:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:35:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:35:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:35:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:35:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:35:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 29 2024-09-14T06:35:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 29 2024-09-14T06:35:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:35:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=12203], [object_number=118, tag_tid=17, entry_tid=12202]]], state=connected] 2024-09-14T06:35:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=12203], [object_number=118, tag_tid=17, entry_tid=12202]]], state=connected] 2024-09-14T06:35:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-09-14T06:35:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:35:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:35:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:35:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:35:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:35:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:35:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:35:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 29 2024-09-14T06:35:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 29 2024-09-14T06:35:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:35:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=12203], [object_number=118, tag_tid=17, entry_tid=12202]]], state=connected] 2024-09-14T06:35:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=12203], [object_number=118, tag_tid=17, entry_tid=12202]]], state=connected] 2024-09-14T06:35:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=117, tag_tid=18, entry_tid=1' 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2024-09-14T06:35:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:35:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:35:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:35:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-14T06:35:10.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 11 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:35:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:33:36 2024 2024-09-14T06:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:34:43 2024 2024-09-14T06:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:35:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap5-clean ']' 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5-clean 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:11.251 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.250+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:35:11.257 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.256+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:11.257 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.256+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:11.311 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:11.309+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:35:11.311 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.310+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:35:11.318 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.317+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:11.388 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.387+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:11.458 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.457+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:35:11.459 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.457+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:35:11.463 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.461+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:11.522 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:11.521+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:35:11.525 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.524+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:11.595 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.594+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:11.598 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.597+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:11.660 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.659+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:35:11.661 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.660+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:35:11.662 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.661+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:11.664 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.663+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:11.669 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.668+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:11.669 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:11.668+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:11.733 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.731+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:35:11.740 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.739+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:11.742 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:11.740+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:35:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6 2024-09-14T06:35:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2024-09-14T06:35:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:35:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:12.828 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:12.827+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:35:12.994 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:12.992+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:12.999 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:12.997+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:13.062 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.061+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:35:13.062 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.061+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:35:13.068 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.067+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:13.136 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:13.135+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:35:13.140 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:13.139+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:13.144 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:13.142+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:13.144 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:13.143+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:35:13.145 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:13.144+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:35:13.201 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:13.199+0000 7f6419d13640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:35:13.276 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.275+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:13.277 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:13.276+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:35:13.348 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.347+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:13.348 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.347+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:13.349 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:13.348+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:35:13.350 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.349+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:13.356 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.355+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:13.484 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.483+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:35:13.485 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.484+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:13.487 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.486+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:13.488 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.487+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:13.488 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:13.487+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:13.555 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:13.554+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:35:13.559 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:13.558+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap6-clean 2024-09-14T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-14T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6-clean 2024-09-14T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:35:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:35:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:35:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:35:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2024-09-14T06:35:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:35:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:35:14.912 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:35:14.909+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-14T06:35:14.912 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:35:14.910+0000 7f5667c8f640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:35:14.913 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:35:14.911+0000 7f566dc9b640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:35:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 21203968 bytes 2024-09-14T06:35:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 38495232 bytes 2024-09-14T06:35:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 43760640 bytes 2024-09-14T06:35:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 83849216 bytes 2024-09-14T06:35:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 84515840 bytes 2024-09-14T06:35:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:35:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:35:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:35:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:35:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:35:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":4325453.47,"entries_behind_primary":972,"entries_per_second":268.03,"non_primary_position":{"entry_tid":6116,"object_number":108,"tag_tid":17},"primary_position":{"entry_tid":7088,"object_number":112,"tag_tid":17},"seconds_until_synced":3} 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:34:51 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:35:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:35:16' 2024-09-14T06:35:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:35:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:35:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:35:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":4325453.47,"entries_behind_primary":972,"entries_per_second":268.03,"non_primary_position":{"entry_tid":6116,"object_number":108,"tag_tid":17},"primary_position":{"entry_tid":7088,"object_number":112,"tag_tid":17},"seconds_until_synced":3} 2024-09-14T06:35:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:35:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:35:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:35:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:35:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:35:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:35:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:35:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:35:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:34:51' 2024-09-14T06:35:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:34:51' 2024-09-14T06:35:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:35:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:34:51' 2024-09-14T06:35:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:35:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:35:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:35:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":4325453.47,"entries_behind_primary":972,"entries_per_second":268.03,"non_primary_position":{"entry_tid":6116,"object_number":108,"tag_tid":17},"primary_position":{"entry_tid":7088,"object_number":112,"tag_tid":17},"seconds_until_synced":3}42624267mirror.0smithi1702024-09-14 06:34:51cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:35:16' 2024-09-14T06:35:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:35:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:35:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:34:51' 2024-09-14T06:35:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:35:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:35:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:35:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:35:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap7 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2024-09-14T06:35:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:35:49.966 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:35:49.964+0000 7f566bc97640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:35:49.966 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:35:49.965+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:35:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:35:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:35:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:35:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:35:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:35:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:35:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:35:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:35:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:35:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:35:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:35:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:35:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:35:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:35:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:35:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:35:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:35:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:35:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:35:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:35:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:35:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:35:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:35:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:35:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:35:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:35:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:35:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:35:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:35:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:35:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:35:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:35:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:35:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:35:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:35:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:35:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:35:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:35:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:35:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:35:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:35:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:35:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:35:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:35:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 31 2024-09-14T06:35:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 31 2024-09-14T06:35:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:35:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=6247], [object_number=126, tag_tid=20, entry_tid=6246]]], state=connected] 2024-09-14T06:35:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=6247], [object_number=126, tag_tid=20, entry_tid=6246]]], state=connected] 2024-09-14T06:35:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=125, tag_tid=21, entry_tid=1' 2024-09-14T06:35:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:35:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:35:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:35:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 31 2024-09-14T06:35:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 31 2024-09-14T06:35:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:35:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=6247], [object_number=126, tag_tid=20, entry_tid=6246]]], state=connected] 2024-09-14T06:35:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=6247], [object_number=126, tag_tid=20, entry_tid=6246]]], state=connected] 2024-09-14T06:35:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:35:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=125, tag_tid=21, entry_tid=1' 2024-09-14T06:35:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=125, tag_tid=21, entry_tid=1' -a 'object_number=125, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=21, entry_tid=1' 2024-09-14T06:35:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-14T06:35:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:35:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2024-09-14T06:35:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:35:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:35:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:35:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-14T06:35:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:35:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 13 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:35:11 2024 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:34:43 2024 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:35:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:35:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap6-clean ']' 2024-09-14T06:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-14T06:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6-clean 2024-09-14T06:35:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:35:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:55.439 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:55.438+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:35:55.520 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.519+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:55.522 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.521+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:55.526 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.524+0000 7f14d881b640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:55.592 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.591+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:35:55.666 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:55.665+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:35:55.669 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:55.668+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:35:55.672 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:55.671+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:55.743 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.742+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:35:55.747 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:55.746+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:55.815 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.814+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:55.815 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:55.814+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:35:55.817 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:55.816+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:35:55.817 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.816+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:55.819 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.818+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:55.824 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:55.823+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:35:55.888 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.887+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:35:55.888 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.887+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:35:55.889 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.887+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:55.957 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.956+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:35:55.963 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.962+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:55.965 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.964+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:55.967 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:55.966+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:56.029 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:56.027+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:35:56.033 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:56.032+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:35:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:57.165 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:57.164+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:35:57.167 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.166+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:35:57.363 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.361+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:57.363 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.362+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:57.430 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.429+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:57.431 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.429+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:35:57.497 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:57.496+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:35:57.497 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:57.496+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:35:57.500 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:57.499+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:35:57.634 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.632+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:35:57.634 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:35:57.632+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:35:57.714 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.713+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:57.793 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.792+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:35:57.800 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:35:57.799+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:35:57.857 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:35:57.856+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:35:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:35:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:35:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:35:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap7-clean 2024-09-14T06:35:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-14T06:35:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7-clean 2024-09-14T06:35:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:35:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:35:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:35:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:35:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:35:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:35:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:35:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2024-09-14T06:35:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:35:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:36:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 46559232 bytes 2024-09-14T06:36:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 99746816 bytes 2024-09-14T06:36:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 146760704 bytes 2024-09-14T06:36:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 174789632 bytes 2024-09-14T06:36:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 236426240 bytes 2024-09-14T06:36:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 299214848 bytes 2024-09-14T06:36:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 322426880 bytes 2024-09-14T06:36:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:36:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:36:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:36:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:36:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:36:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:36:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:36:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:36:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:36:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:36:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:36:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:36:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:36:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:36:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":3212234.13,"entries_behind_primary":0,"entries_per_second":208.3,"non_primary_position":{"entry_tid":6247,"object_number":127,"tag_tid":20},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":19}} 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:35:51 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:36:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:36:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:36:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:36:11' 2024-09-14T06:36:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:36:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:36:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:36:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":3212234.13,"entries_behind_primary":0,"entries_per_second":208.3,"non_primary_position":{"entry_tid":6247,"object_number":127,"tag_tid":20},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":19}} 2024-09-14T06:36:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:36:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:36:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:36:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:36:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:36:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:36:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:36:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:36:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:35:51' 2024-09-14T06:36:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:35:51' 2024-09-14T06:36:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:36:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:35:51' 2024-09-14T06:36:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:36:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:36:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:36:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":3212234.13,"entries_behind_primary":0,"entries_per_second":208.3,"non_primary_position":{"entry_tid":6247,"object_number":127,"tag_tid":20},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":19}}42624267mirror.0smithi1702024-09-14 06:35:51cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:36:11' 2024-09-14T06:36:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:36:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:36:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:35:51' 2024-09-14T06:36:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:36:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:36:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:36:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:36:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8 2024-09-14T06:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap8 2024-09-14T06:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2024-09-14T06:36:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:36:23.827 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.824+0000 7ff305439640 -1 reset not still connected to 0x561ba4420dd0 2024-09-14T06:36:23.827 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.824+0000 7ff305439640 -1 reset not still connected to 0x561ba4421380 2024-09-14T06:36:23.827 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.824+0000 7ff305439640 -1 reset not still connected to 0x561ba4421860 2024-09-14T06:36:23.827 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.825+0000 7ff305439640 -1 reset not still connected to 0x561ba497e270 2024-09-14T06:36:23.828 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.827+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:36:23.828 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.827+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:36:23.828 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.827+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:36:23.828 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.827+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:36:23.828 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.827+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15ba0 2024-09-14T06:36:23.828 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:23.827+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:36:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:36:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:36:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:36:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:36:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:36:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:36:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:36:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:36:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:36:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:36:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:36:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:36:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:36:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:36:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:36:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:36:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:36:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:36:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:36:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:36:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=150, tag_tid=23, entry_tid=19630], [object_number=149, tag_tid=23, entry_tid=19629], [object_number=148, tag_tid=23, entry_tid=19628], [object_number=151, tag_tid=23, entry_tid=19627]]], state=connected] 2024-09-14T06:36:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=150, tag_tid=23, entry_tid=19630], [object_number=149, tag_tid=23, entry_tid=19629], [object_number=148, tag_tid=23, entry_tid=19628], [object_number=151, tag_tid=23, entry_tid=19627]]], state=connected] 2024-09-14T06:36:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=150, tag_tid=23, entry_tid=19630' 2024-09-14T06:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=150, tag_tid=23, entry_tid=19630' 2024-09-14T06:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=150, tag_tid=23, entry_tid=19630' '!=' '' 2024-09-14T06:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=150, tag_tid=23, entry_tid=19630' = '' 2024-09-14T06:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=150, tag_tid=23, entry_tid=19630' 2024-09-14T06:36:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=150, tag_tid=23, entry_tid=19630' 2024-09-14T06:36:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=150, tag_tid=23, entry_tid=19630' 2024-09-14T06:36:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=19630 2024-09-14T06:36:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 19630 2024-09-14T06:36:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=19857], [object_number=148, tag_tid=23, entry_tid=19856], [object_number=151, tag_tid=23, entry_tid=19855], [object_number=150, tag_tid=23, entry_tid=19854]]], state=connected] 2024-09-14T06:36:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=19857], [object_number=148, tag_tid=23, entry_tid=19856], [object_number=151, tag_tid=23, entry_tid=19855], [object_number=150, tag_tid=23, entry_tid=19854]]], state=connected] 2024-09-14T06:36:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=149, tag_tid=23, entry_tid=19857' 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=149, tag_tid=23, entry_tid=19857' 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=149, tag_tid=23, entry_tid=19857' '!=' 'object_number=150, tag_tid=23, entry_tid=19630' 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=149, tag_tid=23, entry_tid=19857' = 'object_number=150, tag_tid=23, entry_tid=19630' 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=149, tag_tid=23, entry_tid=19857' 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=23, entry_tid=19857' 2024-09-14T06:36:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=23, entry_tid=19857' 2024-09-14T06:36:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=19857 2024-09-14T06:36:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 19857 2024-09-14T06:36:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=150, tag_tid=23, entry_tid=20090], [object_number=149, tag_tid=23, entry_tid=20089], [object_number=148, tag_tid=23, entry_tid=20088], [object_number=151, tag_tid=23, entry_tid=20087]]], state=connected] 2024-09-14T06:36:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=150, tag_tid=23, entry_tid=20090], [object_number=149, tag_tid=23, entry_tid=20089], [object_number=148, tag_tid=23, entry_tid=20088], [object_number=151, tag_tid=23, entry_tid=20087]]], state=connected] 2024-09-14T06:36:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=150, tag_tid=23, entry_tid=20090' 2024-09-14T06:36:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=150, tag_tid=23, entry_tid=20090' 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=150, tag_tid=23, entry_tid=20090' '!=' 'object_number=149, tag_tid=23, entry_tid=19857' 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=150, tag_tid=23, entry_tid=20090' = 'object_number=149, tag_tid=23, entry_tid=19857' 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=150, tag_tid=23, entry_tid=20090' 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=150, tag_tid=23, entry_tid=20090' 2024-09-14T06:36:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=150, tag_tid=23, entry_tid=20090' 2024-09-14T06:36:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=20090 2024-09-14T06:36:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 20090 2024-09-14T06:36:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:45.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:45.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=20289], [object_number=148, tag_tid=23, entry_tid=20288], [object_number=151, tag_tid=23, entry_tid=20287], [object_number=150, tag_tid=23, entry_tid=20286]]], state=connected] 2024-09-14T06:36:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=20289], [object_number=148, tag_tid=23, entry_tid=20288], [object_number=151, tag_tid=23, entry_tid=20287], [object_number=150, tag_tid=23, entry_tid=20286]]], state=connected] 2024-09-14T06:36:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=149, tag_tid=23, entry_tid=20289' 2024-09-14T06:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=149, tag_tid=23, entry_tid=20289' 2024-09-14T06:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=149, tag_tid=23, entry_tid=20289' '!=' 'object_number=150, tag_tid=23, entry_tid=20090' 2024-09-14T06:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=149, tag_tid=23, entry_tid=20289' = 'object_number=150, tag_tid=23, entry_tid=20090' 2024-09-14T06:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=149, tag_tid=23, entry_tid=20289' 2024-09-14T06:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=23, entry_tid=20289' 2024-09-14T06:36:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=23, entry_tid=20289' 2024-09-14T06:36:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=20289 2024-09-14T06:36:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 20289 2024-09-14T06:36:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=151, tag_tid=23, entry_tid=20499], [object_number=150, tag_tid=23, entry_tid=20498], [object_number=149, tag_tid=23, entry_tid=20497], [object_number=148, tag_tid=23, entry_tid=20496]]], state=connected] 2024-09-14T06:36:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=151, tag_tid=23, entry_tid=20499], [object_number=150, tag_tid=23, entry_tid=20498], [object_number=149, tag_tid=23, entry_tid=20497], [object_number=148, tag_tid=23, entry_tid=20496]]], state=connected] 2024-09-14T06:36:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=151, tag_tid=23, entry_tid=20499' 2024-09-14T06:36:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=151, tag_tid=23, entry_tid=20499' 2024-09-14T06:36:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=151, tag_tid=23, entry_tid=20499' '!=' 'object_number=149, tag_tid=23, entry_tid=20289' 2024-09-14T06:36:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=151, tag_tid=23, entry_tid=20499' = 'object_number=149, tag_tid=23, entry_tid=20289' 2024-09-14T06:36:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=151, tag_tid=23, entry_tid=20499' 2024-09-14T06:36:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=151, tag_tid=23, entry_tid=20499' 2024-09-14T06:36:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=151, tag_tid=23, entry_tid=20499' 2024-09-14T06:36:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=20499 2024-09-14T06:36:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 20499 2024-09-14T06:36:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=148, tag_tid=23, entry_tid=20704], [object_number=151, tag_tid=23, entry_tid=20703], [object_number=150, tag_tid=23, entry_tid=20702], [object_number=149, tag_tid=23, entry_tid=20701]]], state=connected] 2024-09-14T06:36:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=148, tag_tid=23, entry_tid=20704], [object_number=151, tag_tid=23, entry_tid=20703], [object_number=150, tag_tid=23, entry_tid=20702], [object_number=149, tag_tid=23, entry_tid=20701]]], state=connected] 2024-09-14T06:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=148, tag_tid=23, entry_tid=20704' 2024-09-14T06:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=148, tag_tid=23, entry_tid=20704' 2024-09-14T06:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=148, tag_tid=23, entry_tid=20704' '!=' 'object_number=151, tag_tid=23, entry_tid=20499' 2024-09-14T06:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=148, tag_tid=23, entry_tid=20704' = 'object_number=151, tag_tid=23, entry_tid=20499' 2024-09-14T06:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=148, tag_tid=23, entry_tid=20704' 2024-09-14T06:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=148, tag_tid=23, entry_tid=20704' 2024-09-14T06:36:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=148, tag_tid=23, entry_tid=20704' 2024-09-14T06:36:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=20704 2024-09-14T06:36:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 20704 2024-09-14T06:36:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=148, tag_tid=23, entry_tid=20896], [object_number=151, tag_tid=23, entry_tid=20895], [object_number=150, tag_tid=23, entry_tid=20894], [object_number=149, tag_tid=23, entry_tid=20893]]], state=connected] 2024-09-14T06:36:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=148, tag_tid=23, entry_tid=20896], [object_number=151, tag_tid=23, entry_tid=20895], [object_number=150, tag_tid=23, entry_tid=20894], [object_number=149, tag_tid=23, entry_tid=20893]]], state=connected] 2024-09-14T06:36:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=148, tag_tid=23, entry_tid=20896' 2024-09-14T06:36:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=148, tag_tid=23, entry_tid=20896' 2024-09-14T06:36:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=148, tag_tid=23, entry_tid=20896' '!=' 'object_number=148, tag_tid=23, entry_tid=20704' 2024-09-14T06:36:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=148, tag_tid=23, entry_tid=20896' = 'object_number=148, tag_tid=23, entry_tid=20704' 2024-09-14T06:36:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=148, tag_tid=23, entry_tid=20896' 2024-09-14T06:36:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=148, tag_tid=23, entry_tid=20896' 2024-09-14T06:36:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=148, tag_tid=23, entry_tid=20896' 2024-09-14T06:36:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=20896 2024-09-14T06:36:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 20896 2024-09-14T06:36:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=20925], [object_number=148, tag_tid=23, entry_tid=20924], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=20925], [object_number=148, tag_tid=23, entry_tid=20924], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=149, tag_tid=23, entry_tid=20925' '!=' 'object_number=148, tag_tid=23, entry_tid=20896' 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=149, tag_tid=23, entry_tid=20925' = 'object_number=148, tag_tid=23, entry_tid=20896' 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=24 2024-09-14T06:36:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:36:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=23 2024-09-14T06:36:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:36:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:36:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:36:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=20925 2024-09-14T06:36:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 24 = 23 -a 1 -le 20925 2024-09-14T06:36:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:36:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=20925], [object_number=148, tag_tid=23, entry_tid=20924], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=23, entry_tid=20925], [object_number=148, tag_tid=23, entry_tid=20924], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=149, tag_tid=23, entry_tid=20925' '!=' 'object_number=149, tag_tid=23, entry_tid=20925' 2024-09-14T06:36:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:36:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2024-09-14T06:36:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:36:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:36:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:36:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:36:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:36:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:36:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:36:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:36:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:36:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:36:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:36:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 37 2024-09-14T06:36:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 37 2024-09-14T06:36:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:36:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=149, tag_tid=24, entry_tid=1], [object_number=148, tag_tid=24, entry_tid=0], [object_number=151, tag_tid=23, entry_tid=20923], [object_number=150, tag_tid=23, entry_tid=20922]]], state=connected] 2024-09-14T06:36:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=149, tag_tid=24, entry_tid=1' -a 'object_number=149, tag_tid=24, entry_tid=1' = 'object_number=149, tag_tid=24, entry_tid=1' 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2024-09-14T06:36:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:36:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 15 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:36:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:35:11 2024 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:36:00 2024 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:36:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:36:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap7-clean ']' 2024-09-14T06:36:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-14T06:36:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7-clean 2024-09-14T06:36:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:36:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:36:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:36:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:36:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:50.502 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.501+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:36:50.504 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.503+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:36:50.645 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:50.644+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:36:50.648 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:50.646+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:36:50.652 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:50.651+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:36:50.712 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.710+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:36:50.713 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.712+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:36:50.716 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.714+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:50.717 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.715+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:50.778 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.777+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:36:50.780 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.779+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:50.781 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:50.780+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:50.919 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:36:50.918+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:36:50.920 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:50.919+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:36:50.921 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:36:50.920+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:36:50.922 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:50.920+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:36:50.922 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:50.921+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:36:50.928 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:36:50.927+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:36:50.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:50.996+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:36:51.003 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:51.001+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:36:51.005 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:51.004+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:36:51.007 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:51.005+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:36:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:36:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8 2024-09-14T06:36:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2024-09-14T06:36:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:36:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:36:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:36:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:36:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:52.172 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.171+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:36:52.173 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.171+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:36:52.173 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.171+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:36:52.175 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.174+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:36:52.181 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.180+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:36:52.355 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.354+0000 7f14d881b640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:36:52.363 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.362+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:36:52.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.362+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:36:52.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.363+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:36:52.427 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.425+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:36:52.429 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:36:52.428+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:36:52.505 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.504+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:52.506 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.505+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:52.582 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.580+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:52.586 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.585+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:36:52.653 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:36:52.652+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:36:52.657 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.656+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:36:52.780 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.779+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:36:52.783 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.782+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:36:52.784 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.783+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:36:52.791 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:36:52.790+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:36:52.850 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.849+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:36:52.851 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.850+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:36:52.854 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:36:52.853+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:36:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:36:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap8-clean 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8-clean 2024-09-14T06:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:36:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:36:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:36:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:36:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:36:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:36:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:36:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:36:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:36:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:36:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 24297472 bytes 2024-09-14T06:36:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 82061312 bytes 2024-09-14T06:37:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 126401536 bytes 2024-09-14T06:37:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 178859008 bytes 2024-09-14T06:37:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 202332160 bytes 2024-09-14T06:37:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 223625216 bytes 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:37:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:37:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:37:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:37:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:37:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:37:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:37:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:37:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:37:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:37:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:37:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:37:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:37:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:37:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:37:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":7158336.93,"entries_behind_primary":0,"entries_per_second":447.67,"non_primary_position":{"entry_tid":1,"object_number":149,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":149,"tag_tid":24}} 2024-09-14T06:37:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:36:51 2024-09-14T06:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:37:06' 2024-09-14T06:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:37:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:37:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:37:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":7158336.93,"entries_behind_primary":0,"entries_per_second":447.67,"non_primary_position":{"entry_tid":1,"object_number":149,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":149,"tag_tid":24}} 2024-09-14T06:37:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:37:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:37:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:37:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:37:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:37:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:37:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:37:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:37:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:36:51' 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:36:51' 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:36:51' 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:37:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:37:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":7158336.93,"entries_behind_primary":0,"entries_per_second":447.67,"non_primary_position":{"entry_tid":1,"object_number":149,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":149,"tag_tid":24}}42624267mirror.0smithi1702024-09-14 06:36:51cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:37:06' 2024-09-14T06:37:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:37:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:37:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:36:51' 2024-09-14T06:37:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:37:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:37:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:37:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:37:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap9 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2024-09-14T06:37:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:37:35.696 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:37:35.694+0000 7ff07e6e7640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2024-09-14T06:37:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:37:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:37:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:37:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:37:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:37:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:37:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:37:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:37:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:37:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:37:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:37:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:37:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:37:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:37:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:37:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:37:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:37:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:37:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:37:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:37:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:37:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:37:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:37:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:37:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:37:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:37:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:37:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:37:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:37:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:37:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:37:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:37:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:37:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:37:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:37:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:37:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:37:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:37:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:37:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:37:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:37:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:37:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:37:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:37:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:37:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:37:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:37:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:37:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:37:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:37:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:37:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:37:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:37:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:37:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:37:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:37:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:37:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:37:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:37:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:37:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:37:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:37:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 41 2024-09-14T06:37:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 42 2024-09-14T06:37:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:37:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=27, entry_tid=1], [object_number=168, tag_tid=27, entry_tid=0], [object_number=167, tag_tid=26, entry_tid=15495], [object_number=166, tag_tid=26, entry_tid=15494]]], state=connected] 2024-09-14T06:37:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=169, tag_tid=27, entry_tid=1], [object_number=168, tag_tid=27, entry_tid=0], [object_number=167, tag_tid=26, entry_tid=15495], [object_number=166, tag_tid=26, entry_tid=15494]]], state=connected] 2024-09-14T06:37:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:37:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=169, tag_tid=27, entry_tid=1' 2024-09-14T06:37:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:37:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:37:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:37:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:37:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:37:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:37:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:37:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:37:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:37:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:37:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:37:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:37:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:37:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:37:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 41 2024-09-14T06:37:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 42 2024-09-14T06:37:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:37:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=27, entry_tid=1], [object_number=168, tag_tid=27, entry_tid=0], [object_number=167, tag_tid=26, entry_tid=15495], [object_number=166, tag_tid=26, entry_tid=15494]]], state=connected] 2024-09-14T06:37:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=169, tag_tid=27, entry_tid=1], [object_number=168, tag_tid=27, entry_tid=0], [object_number=167, tag_tid=26, entry_tid=15495], [object_number=166, tag_tid=26, entry_tid=15494]]], state=connected] 2024-09-14T06:37:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=169, tag_tid=27, entry_tid=1' 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=169, tag_tid=27, entry_tid=1' -a 'object_number=169, tag_tid=27, entry_tid=1' = 'object_number=169, tag_tid=27, entry_tid=1' 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:37:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:37:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-14T06:37:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:37:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:37:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:37:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 17 2024-09-14T06:37:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:36:50 2024 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:37:01 2024 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:37:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap8-clean ']' 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8-clean 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:37:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:40.046 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.044+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:37:40.201 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.199+0000 7f14d881b640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:40.203 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.202+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:40.209 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.207+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:37:40.270 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.269+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:40.271 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.270+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:40.333 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.332+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:37:40.335 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.334+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:40.335 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.334+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:40.335 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.334+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:40.335 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.334+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:40.403 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.402+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:37:40.404 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.403+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:37:40.409 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.408+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:40.479 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:37:40.478+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:37:40.485 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.484+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:37:40.546 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.545+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:37:40.548 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.547+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:37:40.627 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.626+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:37:40.629 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.627+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:37:40.631 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.630+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:40.634 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.633+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:40.635 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.634+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:40.705 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:40.704+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:37:40.710 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.709+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:40.714 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:40.712+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:37:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:37:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9 2024-09-14T06:37:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2024-09-14T06:37:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:37:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:37:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:37:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:37:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:41.807 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:41.804+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:37:41.807 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:41.806+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:37:41.808 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:41.806+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:37:41.917 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:37:41.915+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:37:41.917 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:37:41.915+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:37:41.989 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:41.988+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:37:41.991 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:41.990+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:37:41.992 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:41.990+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:41.993 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:41.991+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:37:42.060 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.059+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:37:42.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:42.126+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:37:42.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:42.126+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:37:42.129 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:42.128+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:42.133 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:42.132+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:37:42.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:42.133+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:37:42.271 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.270+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:37:42.275 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:37:42.274+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:37:42.279 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.278+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:37:42.353 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.352+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:42.357 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.356+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:37:42.358 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.357+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:37:42.419 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.418+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:37:42.421 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.420+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:37:42.423 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.422+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:42.429 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.428+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:37:42.501 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.500+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:37:42.567 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:37:42.566+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:37:42.568 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:37:42.567+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:37:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:37:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:37:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap9-clean 2024-09-14T06:37:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-14T06:37:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:37:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9-clean 2024-09-14T06:37:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:37:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:37:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T06:37:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:37:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:37:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2024-09-14T06:37:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:37:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T06:37:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 19661824 bytes 2024-09-14T06:37:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 105005056 bytes 2024-09-14T06:37:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 167927808 bytes 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:37:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:37:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:37:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:37:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local description_pattern= 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local service_pattern= 2024-09-14T06:37:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:37:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:37:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T06:37:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='test: 2024-09-14T06:37:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: global_id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:37:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+replaying 2024-09-14T06:37:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: replaying, {"bytes_per_second":3696922.53,"entries_behind_primary":1,"entries_per_second":233.53,"non_primary_position":{"entry_tid":1,"object_number":169,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":169,"tag_tid":28},"seconds_until_synced":0} 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: service: mirror.0 on smithi170 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:37:51 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: peer_sites: 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: name: cluster2 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: state: up+stopped 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: description: local image is primary 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: last_update: 2024-09-14 06:37:51' 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T06:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: state: up+replaying 2024-09-14T06:37:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^ description: .*' 2024-09-14T06:37:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: description: replaying, {"bytes_per_second":3696922.53,"entries_behind_primary":1,"entries_per_second":233.53,"non_primary_position":{"entry_tid":1,"object_number":169,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":169,"tag_tid":28},"seconds_until_synced":0} 2024-09-14T06:37:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:37:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:37:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep '^up+' 2024-09-14T06:37:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:37:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T06:37:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: service: mirror.0 on smithi170 2024-09-14T06:37:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update 2024-09-14T06:37:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T06:37:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:37:51' 2024-09-14T06:37:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 06:37:51' 2024-09-14T06:37:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:37:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state_pattern=up+replaying 2024-09-14T06:37:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'prev_last_update=2024-09-14 06:37:51' 2024-09-14T06:37:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status 2024-09-14T06:37:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ CEPH_ARGS= 2024-09-14T06:37:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T06:37:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status='OKOKOK142624267mirror.0smithi17019.3.0-4775-g7f24e589trueOK42694272mirror.1smithi17019.3.0-4775-g7f24e589falseOKtest8dc8de06-79f6-41b0-809d-df33a6a60e27up+replayingreplaying, {"bytes_per_second":3696922.53,"entries_behind_primary":1,"entries_per_second":233.53,"non_primary_position":{"entry_tid":1,"object_number":169,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":169,"tag_tid":28},"seconds_until_synced":0}42624267mirror.0smithi1702024-09-14 06:37:51cluster2c3b2353a-f43b-43f4-9abd-f7c6b12eaa5aup+stoppedlocal image is primary2024-09-14 06:37:51' 2024-09-14T06:37:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_update state 2024-09-14T06:37:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T06:37:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_update='2024-09-14 06:37:51' 2024-09-14T06:37:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T06:37:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=up+replaying 2024-09-14T06:37:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo up+replaying 2024-09-14T06:37:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep up+replaying 2024-09-14T06:37:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:up+replaying 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap10 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2024-09-14T06:37:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:38:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:38:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:38:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:38:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:38:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:38:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:38:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:38:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:38:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:38:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:38:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:38:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:38:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:38:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:38:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:38:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:38:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:38:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:38:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:38:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:38:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:38:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:38:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:38:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:38:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:38:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:38:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:38:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:38:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:38:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=182, tag_tid=29, entry_tid=12106], [object_number=181, tag_tid=29, entry_tid=12105], [object_number=180, tag_tid=29, entry_tid=12104], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:38:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=182, tag_tid=29, entry_tid=12106], [object_number=181, tag_tid=29, entry_tid=12105], [object_number=180, tag_tid=29, entry_tid=12104], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=181, tag_tid=30, entry_tid=1' -a 'object_number=181, tag_tid=30, entry_tid=1' = 'object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=182, tag_tid=29, entry_tid=12106' '!=' '' 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=182, tag_tid=29, entry_tid=12106' = '' 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:38:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=30 2024-09-14T06:38:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:38:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=29 2024-09-14T06:38:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:38:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:38:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:38:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=12106 2024-09-14T06:38:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 30 = 29 -a 1 -le 12106 2024-09-14T06:38:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:38:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:38:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:38:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:38:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:38:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:38:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:38:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:38:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:38:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:38:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:38:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:38:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=182, tag_tid=29, entry_tid=12106], [object_number=181, tag_tid=29, entry_tid=12105], [object_number=180, tag_tid=29, entry_tid=12104], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:38:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=182, tag_tid=29, entry_tid=12106], [object_number=181, tag_tid=29, entry_tid=12105], [object_number=180, tag_tid=29, entry_tid=12104], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=181, tag_tid=30, entry_tid=1' -a 'object_number=181, tag_tid=30, entry_tid=1' = 'object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=182, tag_tid=29, entry_tid=12106' '!=' 'object_number=182, tag_tid=29, entry_tid=12106' 2024-09-14T06:38:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:38:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2024-09-14T06:38:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:38:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:38:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:38:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:38:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:38:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:38:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:38:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:38:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:38:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:38:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=181, tag_tid=30, entry_tid=1' -a 'object_number=181, tag_tid=30, entry_tid=1' = 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:38:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:38:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-14T06:38:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 19 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:36:50 2024 2024-09-14T06:38:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:38:02 2024 2024-09-14T06:38:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:38:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:38:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:38:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:38:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:38:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:38:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n snap9-clean ']' 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9-clean 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:29.050 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.049+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:29.136 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.135+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:29.280 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.279+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:29.281 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.279+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:29.282 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.280+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:29.290 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.289+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:29.296 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.294+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:29.350 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.348+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:29.352 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.351+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:29.355 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.354+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:29.416 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.414+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:29.419 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.418+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:29.494 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:29.493+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:29.495 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:29.493+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:29.495 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.493+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:29.563 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:29.562+0000 7f641f51e640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:38:29.564 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:29.562+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:38:29.564 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:29.563+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:38:29.564 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.563+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:29.565 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:29.564+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:38:29.567 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.566+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:29.568 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.566+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:29.568 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:29.567+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:29.574 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.573+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:29.646 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.645+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:29.654 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.653+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:29.656 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.655+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:29.656 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.655+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:29.703 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.702+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:29.771 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.770+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:29.772 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:29.771+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:29.779 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:29.776+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap10 2024-09-14T06:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2024-09-14T06:38:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:30.846 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:30.844+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:30.848 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:30.844+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:30.966 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:30.964+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:30.966 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:30.964+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:38:30.967 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:30.965+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:38:30.967 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:30.965+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:31.040 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.038+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:31.042 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.040+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:31.044 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.042+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:31.047 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.044+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:31.101 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.099+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:31.103 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.101+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:31.106 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.104+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:31.110 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.108+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:31.168 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.166+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:31.171 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.169+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:31.176 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.174+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:31.241 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:31.239+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:38:31.245 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:31.243+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:31.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:31.244+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:31.312 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.310+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:31.314 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:31.312+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:31.318 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:31.316+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:38:31.319 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:31.317+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:38:31.376 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.374+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:31.376 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.374+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:31.377 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.375+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:31.381 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.378+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:31.384 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.381+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:31.453 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.450+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:31.457 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.455+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:31.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.458+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:31.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.459+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:31.519 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.517+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:31.520 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:31.518+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:31.522 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.520+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:31.526 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:31.524+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:31.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ clean_snap_name=snap10-clean 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10-clean 2024-09-14T06:38:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T06:38:32.015 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:38:32.013+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:38:32.016 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:38:32.015+0000 7f566a494640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:38:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:38:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:38:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T06:38:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:38:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:38:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:38:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T06:38:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:38:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:38:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:38:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:38:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:38:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:38:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:38:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T06:38:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:38:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:38:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:38:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:38:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:38:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:38:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:38:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:38:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:38:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:38:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:38:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:38:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:38:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:38:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:38:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:38:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T06:38:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:38:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:38:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:38:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:38:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:38:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:38:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:38:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:38:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:38:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:38:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:38:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:38:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:38:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:38:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:38:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:38:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:38:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:38:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:38:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:38:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:38:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:38:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:38:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:38:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:38:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:38:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:38:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:38:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:38:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:38:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:38:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:38:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=181, tag_tid=31, entry_tid=1' 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:38:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=181, tag_tid=31, entry_tid=1' -a 'object_number=181, tag_tid=31, entry_tid=1' = 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=181, tag_tid=30, entry_tid=1' '!=' '' 2024-09-14T06:38:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:38:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=181, tag_tid=30, entry_tid=1' = '' 2024-09-14T06:38:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=181, tag_tid=31, entry_tid=1' 2024-09-14T06:38:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:38:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=31 2024-09-14T06:38:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:38:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=30 2024-09-14T06:38:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=181, tag_tid=31, entry_tid=1' 2024-09-14T06:38:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:38:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:38:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:38:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:38:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=1 2024-09-14T06:38:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 31 = 30 -a 1 -le 1 2024-09-14T06:38:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:38:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:38:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:38:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:38:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:38:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:38:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=181, tag_tid=31, entry_tid=1' 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=30, entry_tid=1], [object_number=180, tag_tid=30, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=181, tag_tid=31, entry_tid=1' -a 'object_number=181, tag_tid=31, entry_tid=1' = 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=181, tag_tid=30, entry_tid=1' '!=' 'object_number=181, tag_tid=30, entry_tid=1' 2024-09-14T06:38:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:38:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.4 2024-09-14T06:38:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror test 2024-09-14T06:38:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n test ']' 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:38:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:38:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:38:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:38:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:38:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:38:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:38:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:38:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:38:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:38:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:38:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:38:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:38:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:38:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:38:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:38:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:38:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:38:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:38:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-14T06:38:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=181, tag_tid=31, entry_tid=1' 2024-09-14T06:38:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-14T06:38:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-14T06:38:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:38:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:38:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:38:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:38:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-14T06:38:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-test.status 2024-09-14T06:38:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:38:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-14T06:38:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 45 2024-09-14T06:38:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 45 2024-09-14T06:38:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:38:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=181, tag_tid=31, entry_tid=1], [object_number=180, tag_tid=31, entry_tid=0], [object_number=182, tag_tid=29, entry_tid=12106], [object_number=183, tag_tid=29, entry_tid=12103]]], state=connected] 2024-09-14T06:38:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.status 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=181, tag_tid=31, entry_tid=1' 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=181, tag_tid=31, entry_tid=1' -a 'object_number=181, tag_tid=31, entry_tid=1' = 'object_number=181, tag_tid=31, entry_tid=1' 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10-clean 2024-09-14T06:38:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:38:35.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:38:35.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:38:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-14T06:38:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'test': 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 20 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab9a7c50fd 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab9a7c50fd 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:30:23 2024 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:38:29 2024 2024-09-14T06:38:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:38:02 2024 2024-09-14T06:38:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:38:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab9a7c50fd 2024-09-14T06:38:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:38:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:38:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8dc8de06-79f6-41b0-809d-df33a6a60e27 2024-09-14T06:38:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:38:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2024-09-14T06:38:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:37.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:37.132+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:37.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:37.132+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:37.141 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:37.138+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:37.334 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:37.331+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:37.334 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:37.332+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:37.337 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:37.335+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:37.338 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:37.336+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:37.394 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:37.392+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1-clean 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap1-clean 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:38.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:38.486 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:38.485+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:38.487 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:38.486+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:38.487 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:38.486+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:38.571 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.569+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:38.572 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.570+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:38.575 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.574+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:38.575 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.574+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:38.578 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.577+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:38.581 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.580+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:38.782 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:38.781+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:38.784 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:38.783+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:38.787 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:38.786+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:38.834 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.832+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:38.834 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:38.833+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:39.992 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:39.990+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:39.992 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:39.991+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:40.055 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:40.053+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:40.057 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:40.055+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:40.058 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:40.057+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:40.063 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:40.061+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:40.168 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:40.167+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:40.170 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:40.169+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:40.217 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:40.215+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:40.341 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:40.340+0000 7f641f51e640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:40.341 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:40.340+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:40.346 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:40.345+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:38:40.455 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:40.454+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:40.456 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:40.455+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:40.517 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:40.515+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:40.520 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:40.519+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:40.524 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:40.523+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2-clean 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap2-clean 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:41.705 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:41.704+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:41.706 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:41.705+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:41.708 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:41.707+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:41.711 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:41.710+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:41.712 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:41.711+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:41.714 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:41.713+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:41.810 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:41.809+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:41.813 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:41.812+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:41.878 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:41.877+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:38:41.880 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:41.879+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:41.941 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:41.940+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:41.942 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:41.940+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:41.946 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:41.945+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:38:42.042 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:42.040+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:42.042 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:42.040+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:42.042 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:42.040+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:42.102 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:42.100+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:42.106 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:42.105+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:42.108 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:42.107+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:42.109 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:42.108+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:43.272 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:43.271+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:43.274 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:43.273+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:43.278 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:43.277+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:43.393 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:43.392+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:43.397 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:43.396+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:43.398 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:43.397+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:43.526 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:43.525+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:43.589 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:43.588+0000 7f641f51e640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:38:43.592 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:43.591+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:43.597 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:43.595+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:43.597 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:43.595+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:43.597 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:43.596+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:43.772 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:43.771+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:43.776 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:43.774+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3-clean 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap3-clean 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:44.955 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:44.954+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:44.960 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:44.959+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:44.960 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:44.959+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:45.065 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:45.063+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:45.066 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:45.065+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:45.066 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:45.065+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:45.113 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:45.112+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:45.256 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:45.253+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:45.311 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:45.310+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:45.312 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:45.310+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:45.314 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:45.313+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4 2024-09-14T06:38:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4 2024-09-14T06:38:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4 2024-09-14T06:38:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:46.579 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:46.577+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:46.646 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:46.644+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:46.702 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:46.701+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:46.705 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:46.704+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:46.706 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:46.704+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:46.706 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:46.705+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:46.845 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:46.844+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:38:46.850 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:46.849+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:46.853 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:46.852+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:47.030 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:47.029+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:47.031 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:47.030+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:47.034 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:47.033+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:47.039 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:47.037+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4-clean 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap4-clean 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:48.097 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:48.096+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:48.234 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:48.233+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:48.240 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:48.239+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:48.241 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:48.240+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:48.357 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:48.356+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:48.357 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:48.356+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:48.364 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:48.363+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:48.369 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:48.367+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:48.492 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:48.490+0000 7f641f51e640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:38:48.492 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:48.490+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:38:48.494 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:48.492+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:48.611 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:48.610+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:38:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:49.696 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.695+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:49.697 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:49.694+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:49.697 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:49.695+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:49.697 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:49.696+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:49.698 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.696+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:49.698 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.696+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:49.806 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:49.804+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:49.811 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:49.809+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:49.874 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.873+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:49.874 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.873+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:49.877 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.875+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:49.880 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.879+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:49.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.882+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:49.938 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:49.937+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:49.945 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:49.944+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:49.952 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:49.951+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:50.008 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:50.006+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:50.063 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:50.062+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:38:50.066 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:50.065+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:38:50.130 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.128+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:50.130 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.129+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:50.130 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.129+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:50.131 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.130+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:50.134 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.133+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:50.137 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:50.135+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:50.206 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.205+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:50.211 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.209+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:50.271 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.270+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:50.278 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.277+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:50.278 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.277+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:50.279 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:50.278+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:50.348 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:50.346+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:50.349 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:50.347+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5-clean 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap5-clean 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:51.418 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:51.416+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:51.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:51.417+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:51.421 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:51.420+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:51.600 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:51.598+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:51.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:51.599+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:51.664 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:51.663+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:51.735 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:51.734+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:51.744 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:51.742+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:51.744 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:51.742+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:51.816 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:51.815+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:51.823 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:51.821+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:51.887 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:51.886+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:38:51.889 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:51.888+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:51.895 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:51.894+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:51.900 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:51.899+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:52.017 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:52.015+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:52.017 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:52.016+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:52.020 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:52.019+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:52.087 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:52.086+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:52.088 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:52.087+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:53.184 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.183+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:53.185 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.184+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:38:53.186 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.185+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:53.189 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.188+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:53.189 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.188+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:53.458 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.457+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:53.517 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.516+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:53.522 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.521+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:53.525 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.524+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:53.589 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.587+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:53.664 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.663+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:53.668 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.667+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:53.669 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:53.668+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:53.740 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:53.738+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:53.747 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:53.746+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:38:53.805 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.804+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:53.805 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.804+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:53.809 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:53.808+0000 7f641f51e640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:53.813 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:53.811+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 6 2024-09-14T06:38:53.864 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.863+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:53.871 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.870+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:53.873 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.872+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:53.936 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.935+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:53.942 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.940+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:53.943 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:53.941+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:54.010 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:54.008+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:54.010 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:54.009+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:54.016 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:54.014+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6-clean 2024-09-14T06:38:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-14T06:38:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap6-clean 2024-09-14T06:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:55.116 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.113+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:55.116 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.114+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:55.116 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.115+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:55.117 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.116+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:55.118 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.117+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:55.501 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.497+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:55.502 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.499+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:55.506 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.505+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:55.507 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.506+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:55.640 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.638+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:38:55.640 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.639+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:55.643 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.642+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:55.644 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.642+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:55.646 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.645+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:55.648 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.647+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:55.716 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:55.715+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:55.780 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:55.779+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:38:55.781 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:55.779+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:38:55.783 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.782+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:55.856 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.854+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:55.856 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.855+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:55.917 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.916+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:55.925 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.923+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:55.926 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.924+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:55.928 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.926+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:55.994 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.993+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:55.999 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:55.997+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:56.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:57.126 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.124+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:57.126 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.124+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:57.127 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.124+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:57.127 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.126+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:57.285 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.283+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:57.291 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.289+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:57.292 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.290+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:57.361 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.360+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:57.366 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.364+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:57.425 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.424+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:57.428 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.426+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:57.428 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.426+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:57.431 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.430+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:57.432 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.431+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:57.541 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.539+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:57.546 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:57.545+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:57.630 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.629+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:57.632 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.631+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:57.706 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.704+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:57.712 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.711+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:57.713 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:57.711+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:38:57.773 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.772+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:57.775 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.774+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:57.779 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:57.778+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:38:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7-clean 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap7-clean 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:58.877 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:58.875+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:58.877 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:58.876+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:58.877 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:58.876+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:58.879 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:58.878+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:58.879 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:58.878+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:38:59.049 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.048+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:59.050 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.048+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:59.053 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.052+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:59.055 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.054+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:59.058 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.057+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:59.121 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.119+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:59.123 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.122+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:59.195 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:59.194+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:38:59.196 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:59.195+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:38:59.262 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:38:59.260+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:38:59.264 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.263+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:38:59.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.267+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:59.328 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.326+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:59.330 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:59.329+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:38:59.330 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.329+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:59.336 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:59.335+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:38:59.393 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.392+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:38:59.394 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.393+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:38:59.395 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.394+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:38:59.397 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.396+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:59.399 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:38:59.398+0000 7f641f51e640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:38:59.400 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:38:59.399+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:38:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:38:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:38:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:38:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8 2024-09-14T06:38:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:38:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:00.752 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:00.751+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:39:00.754 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:00.753+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:00.754 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:00.753+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:00.763 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:00.762+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:00.828 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:00.827+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:00.835 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:00.833+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:00.836 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:00.835+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:00.897 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:00.895+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:00.901 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:00.900+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:00.971 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:00.970+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:00.973 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:00.972+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:01.049 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:01.046+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:01.114 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.112+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:39:01.122 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.121+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:01.189 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.187+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:39:01.193 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.191+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:01.198 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.197+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:01.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.267+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:39:01.269 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.267+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:39:01.276 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.274+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:01.276 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.275+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:01.277 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:01.276+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:01.339 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:01.337+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:01.339 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:01.338+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8-clean 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap8-clean 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:02.425 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:02.423+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:39:02.429 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:02.428+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:02.429 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:02.428+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:02.461 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:02.460+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:39:02.631 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:02.629+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:02.793 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:02.792+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:39:02.793 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:02.792+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:39:02.907 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:02.906+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:39:02.980 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:02.979+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:02.981 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:02.979+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:39:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:04.093 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.092+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:04.125 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.124+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:04.125 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.124+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:04.126 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.124+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:04.259 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.258+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:04.259 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.258+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:04.264 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.263+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:04.265 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.263+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:04.325 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.322+0000 7ff2f4806640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:04.325 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.323+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:04.327 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.326+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:04.327 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.326+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:04.332 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.331+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:39:04.388 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.386+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:04.392 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.390+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:04.392 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.391+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:04.501 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.500+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:04.560 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:04.559+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:39:04.561 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:04.560+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:39:04.572 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.570+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:04.572 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.570+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:04.639 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.637+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:04.643 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.642+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:04.645 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.643+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:04.707 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.706+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:39:04.777 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.776+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:04.778 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:04.777+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:04.783 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:04.781+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9-clean 2024-09-14T06:39:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-14T06:39:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap9-clean 2024-09-14T06:39:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:39:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:05.856 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:05.855+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:39:05.856 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:05.855+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:05.856 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:05.855+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:39:05.950 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:05.949+0000 7ff2f3003640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:06.023 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.022+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:39:06.023 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.022+0000 7f14d881b640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:39:06.025 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.023+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:06.025 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.024+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:06.087 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:06.086+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:39:06.091 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.090+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:06.098 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.096+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:39:06.098 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.096+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:39:06.161 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.160+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:06.163 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.162+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:06.169 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.167+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:06.171 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.169+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:39:06.307 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:06.305+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:39:06.310 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.309+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:06.377 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.374+0000 7f14d7819640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:39:06.378 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.376+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:06.379 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.378+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:06.380 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.379+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:06.386 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.385+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:06.446 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.445+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:39:06.449 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.447+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:06.449 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.448+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:06.451 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.450+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:06.517 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:06.516+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:39:06.522 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:06.521+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:06.590 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.589+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:06.591 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:06.590+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10 2024-09-14T06:39:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap10 2024-09-14T06:39:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10 2024-09-14T06:39:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:39:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:07.684 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.683+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:07.687 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.686+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:07.786 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:07.784+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:39:07.786 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:07.785+0000 7f641c518640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:39:07.846 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.844+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:07.849 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:07.848+0000 7f14d881b640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:07.849 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:07.848+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:07.917 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:07.916+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:07.925 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:07.924+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:07.926 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:07.924+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:07.986 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.985+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:07.987 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.986+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:07.987 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.986+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:07.992 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.991+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:07.994 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:07.993+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:08.053 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.052+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:08.055 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.054+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:08.055 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.054+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:08.057 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.056+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:08.060 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.059+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:08.062 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.061+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:39:08.136 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:08.135+0000 7f641e51c640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:39:08.136 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:08.135+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:39:08.138 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:08.137+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:39:08.141 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.140+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:08.213 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:08.212+0000 7f641bd17640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:39:08.216 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:08.214+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:08.220 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:08.218+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:08.222 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:08.220+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:08.351 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:08.350+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:39:08.352 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:08.350+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:39:08.352 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:08.350+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:39:08.352 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:08.351+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:39:08.355 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:08.354+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:39:08.422 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.420+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:39:08.427 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.425+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:08.427 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.426+0000 7ff2f07fe640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:08.427 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:08.426+0000 7ff2f1800640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10-clean 2024-09-14T06:39:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-14T06:39:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap10-clean 2024-09-14T06:39:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:39:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export 2024-09-14T06:39:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:09.516 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.514+0000 7ff2f700b640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:09.516 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.515+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:09.518 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.517+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:09.518 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.517+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:09.677 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.674+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:09.677 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.675+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:09.678 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.676+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:09.678 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.676+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:09.747 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.746+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:09.747 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.746+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:09.747 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.746+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:09.811 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.810+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:09.812 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.811+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:39:09.814 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.813+0000 7ff2f0fff640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:39:09.817 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.816+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-09-14T06:39:09.823 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.822+0000 7ff2f780c640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-09-14T06:39:09.882 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:39:09.881+0000 7ff2f3804640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:39:09.888 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.887+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:09.957 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:09.956+0000 7f641a514640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-14T06:39:09.959 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:09.958+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:39:09.961 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:39:09.960+0000 7f641ad15640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:39:09.966 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:09.965+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:10.041 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:10.040+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:10.047 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:10.044+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:10.110 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:10.109+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:10.114 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:10.113+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 6 2024-09-14T06:39:10.175 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:10.174+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:39:10.177 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:39:10.176+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:39:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T06:39:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-test.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-test.export 2024-09-14T06:39:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2024-09-14T06:39:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1 2024-09-14T06:39:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-14T06:39:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap1 2024-09-14T06:39:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-14T06:39:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap1-clean 2024-09-14T06:39:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-14T06:39:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap1-clean 2024-09-14T06:39:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-14T06:39:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2 2024-09-14T06:39:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-14T06:39:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap2 2024-09-14T06:39:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-14T06:39:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap2-clean 2024-09-14T06:39:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-14T06:39:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap2-clean 2024-09-14T06:39:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-14T06:39:13.921 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:13.919+0000 7f566cc99640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:39:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3 2024-09-14T06:39:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-14T06:39:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap3 2024-09-14T06:39:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-14T06:39:14.950 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:14.948+0000 7f566cc99640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:39:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap3-clean 2024-09-14T06:39:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-14T06:39:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap3-clean 2024-09-14T06:39:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-14T06:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4 2024-09-14T06:39:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-14T06:39:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap4 2024-09-14T06:39:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-14T06:39:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap4-clean 2024-09-14T06:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-14T06:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap4-clean 2024-09-14T06:39:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-14T06:39:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap5 2024-09-14T06:39:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-14T06:39:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap5-clean 2024-09-14T06:39:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-14T06:39:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap5-clean 2024-09-14T06:39:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-14T06:39:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6 2024-09-14T06:39:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-14T06:39:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap6 2024-09-14T06:39:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-14T06:39:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap6-clean 2024-09-14T06:39:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-14T06:39:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap6-clean 2024-09-14T06:39:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-14T06:39:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7 2024-09-14T06:39:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-14T06:39:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap7 2024-09-14T06:39:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-14T06:39:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap7-clean 2024-09-14T06:39:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-14T06:39:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap7-clean 2024-09-14T06:39:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-14T06:39:24.094 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:24.093+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:39:24.097 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:24.096+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:39:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap8 2024-09-14T06:39:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-14T06:39:25.132 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:25.131+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-14T06:39:25.133 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:25.131+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-14T06:39:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap8-clean 2024-09-14T06:39:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-14T06:39:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap8-clean 2024-09-14T06:39:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-14T06:39:26.109 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:26.108+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-14T06:39:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9 2024-09-14T06:39:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-14T06:39:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap9 2024-09-14T06:39:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-14T06:39:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap9-clean 2024-09-14T06:39:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-14T06:39:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap9-clean 2024-09-14T06:39:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-14T06:39:28.119 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:28.117+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-14T06:39:28.119 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:28.118+0000 7f566cc99640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:39:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap10 2024-09-14T06:39:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-14T06:39:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap10-clean 2024-09-14T06:39:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-14T06:39:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap10-clean 2024-09-14T06:39:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-14T06:39:30.086 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:30.085+0000 7f566a494640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:39:30.088 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:39:30.086+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:39:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:39:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror test 2024-09-14T06:39:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:39:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:39:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror test 2024-09-14T06:39:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-14T06:39:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-14T06:39:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:39:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:39:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-14T06:39:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:39:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-14T06:39:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:39:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:39:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:39:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-14T06:39:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=test 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror test 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=test 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-14T06:39:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:39:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-14T06:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: create many images' 2024-09-14T06:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.RQ7U70XNbJ/rbd-mirror.test.log 2024-09-14T06:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2024-09-14T06:39:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2024-09-14 06:39:31 TEST: create many images 2024-09-14T06:39:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14 06:39:31 TEST: create many images 2024-09-14T06:39:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ snap_name=snap 2024-09-14T06:39:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:39:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-09-14T06:39:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:39:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:39:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2024-09-14T06:39:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:39:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:39:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 54477824 bytes 2024-09-14T06:39:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 101121024 bytes 2024-09-14T06:39:38.757 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:39:38.756+0000 7ff07d6e5640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-09-14T06:39:38.761 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:39:38.760+0000 7ff081eee640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-09-14T06:39:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 114785280 bytes 2024-09-14T06:39:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 135950336 bytes 2024-09-14T06:39:42.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 181815296 bytes 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:39:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:39:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:39:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:39:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-09-14T06:39:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:39:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:39:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:39:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:39:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:39:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-14T06:39:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:39:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:39:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:39:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:39:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:39:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 55894016 bytes 2024-09-14T06:39:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 78727168 bytes 2024-09-14T06:39:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 172068864 bytes 2024-09-14T06:39:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 204285952 bytes 2024-09-14T06:39:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 227565568 bytes 2024-09-14T06:39:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:39:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:39:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:39:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:39:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:39:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2024-09-14T06:39:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2024-09-14T06:39:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:39:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:39:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:39:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-14T06:39:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:39:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:39:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:39:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:39:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:39:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-09-14T06:39:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:39:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:39:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:39:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:39:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:39:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-14T06:39:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:39:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:39:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:39:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:39:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2024-09-14T06:39:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:39:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:39:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 22677504 bytes 2024-09-14T06:39:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 64148480 bytes 2024-09-14T06:40:00.584 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.582+0000 7ff305439640 -1 reset not still connected to 0x561ba4421380 2024-09-14T06:40:00.661 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.582+0000 7ff305439640 -1 reset not still connected to 0x561ba4421860 2024-09-14T06:40:00.662 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.584+0000 7ff305439640 -1 reset not still connected to 0x561ba497e270 2024-09-14T06:40:00.662 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.584+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:40:00.662 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.584+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:40:00.662 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.584+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:40:00.663 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.585+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:40:00.663 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.585+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:40:00.663 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:40:00.585+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:40:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 80652288 bytes 2024-09-14T06:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 93456384 bytes 2024-09-14T06:40:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 94070784 bytes 2024-09-14T06:40:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:40:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:40:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:40:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:40:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-09-14T06:40:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:40:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:40:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:40:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:40:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:40:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-14T06:40:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:40:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:40:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2024-09-14T06:40:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:40:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:40:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 49073152 bytes 2024-09-14T06:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 65133568 bytes 2024-09-14T06:40:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 144239616 bytes 2024-09-14T06:40:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:40:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:40:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-09-14T06:40:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:40:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:40:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:40:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:40:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:40:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-14T06:40:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:40:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:40:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:40:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:40:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:40:17.713 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:40:17.712+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:40:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 59457536 bytes 2024-09-14T06:40:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 110210048 bytes 2024-09-14T06:40:22.346 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:22.345+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:40:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 146434048 bytes 2024-09-14T06:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 147075072 bytes 2024-09-14T06:40:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 168080384 bytes 2024-09-14T06:40:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:40:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:40:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:40:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:40:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:40:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2024-09-14T06:40:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:40:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:40:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:40:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:40:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:40:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:40:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2024-09-14T06:40:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:40:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:40:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 15724544 bytes 2024-09-14T06:40:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 34729984 bytes 2024-09-14T06:40:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 127900672 bytes 2024-09-14T06:40:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 128538624 bytes 2024-09-14T06:40:35.003 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:35.001+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:40:35.003 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:35.002+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:40:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 159387648 bytes 2024-09-14T06:40:36.615 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:36.614+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:40:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 230991872 bytes 2024-09-14T06:40:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:40:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:40:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:40:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:40:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-14T06:40:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:40:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:40:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:40:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:40:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-09-14T06:40:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:40:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:40:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2024-09-14T06:40:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:40:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:40:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 58778624 bytes 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:40:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:40:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:40:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:40:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:40:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:40:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2024-09-14T06:40:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:40:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:40:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 83642368 bytes 2024-09-14T06:40:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:40:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:40:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:40:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:40:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:40:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2024-09-14T06:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:40:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 37072896 bytes 2024-09-14T06:40:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 57030656 bytes 2024-09-14T06:40:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 63174656 bytes 2024-09-14T06:40:58.771 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:58.769+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:40:58.774 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:58.772+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:40:58.775 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:58.772+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:40:58.775 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:40:58.772+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:40:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 76720128 bytes 2024-09-14T06:41:01.691 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:01.687+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:41:01.691 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:01.689+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:41:01.691 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:01.689+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:41:01.691 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:01.689+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:41:01.972 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:41:01.970+0000 7f0c5ac0d640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 6 2024-09-14T06:41:01.974 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:41:01.973+0000 7f0c57406640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 7 2024-09-14T06:41:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 130896896 bytes 2024-09-14T06:41:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 178368512 bytes 2024-09-14T06:41:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:41:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:41:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:41:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-09-14T06:41:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:41:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:41:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:41:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:41:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:41:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-14T06:41:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:41:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:41:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2024-09-14T06:41:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:41:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:41:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 50745344 bytes 2024-09-14T06:41:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 122796032 bytes 2024-09-14T06:41:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:41:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:41:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:41:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2024-09-14T06:41:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:41:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:41:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2024-09-14T06:41:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:41:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:41:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 10297344 bytes 2024-09-14T06:41:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 37803008 bytes 2024-09-14T06:41:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 38412288 bytes 2024-09-14T06:41:19.388 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.387+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:41:19.391 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.390+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:41:19.391 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.390+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:41:19.574 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561ba497e270 2024-09-14T06:41:19.574 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:41:19.574 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:41:19.574 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:41:19.575 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:41:19.575 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:41:19.575 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:41:19.575 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:41:19.575 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.572+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:41:19.853 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.851+0000 7ff305439640 -1 reset not still connected to 0x561ba4420d00 2024-09-14T06:41:19.853 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.851+0000 7ff305439640 -1 reset not still connected to 0x561ba5a149c0 2024-09-14T06:41:19.853 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.851+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:41:19.853 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.851+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:41:19.853 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.851+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:41:19.856 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:19.854+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:41:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 85256192 bytes 2024-09-14T06:41:21.523 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:41:21.520+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b450 2024-09-14T06:41:21.523 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:41:21.520+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:41:21.523 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:41:21.520+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:41:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:41:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:41:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:41:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:41:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:41:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:41:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:41:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:41:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:41:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:41:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:41:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2024-09-14T06:41:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:41:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:41:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 4997120 bytes 2024-09-14T06:41:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 56544256 bytes 2024-09-14T06:41:26.278 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:26.276+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:41:26.278 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:26.277+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:41:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 124615680 bytes 2024-09-14T06:41:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:41:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:41:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:41:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:41:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:41:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-14T06:41:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-14T06:41:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:41:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:41:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2024-09-14T06:41:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:41:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:41:33.016 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:41:33.015+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:41:33.017 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:41:33.015+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:41:34.845 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:34.843+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:41:34.848 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:34.845+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:41:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 85664768 bytes 2024-09-14T06:41:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:41:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:41:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:41:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2024-09-14T06:41:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:41:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:41:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:41:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:41:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:41:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-14T06:41:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:41:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:41:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2024-09-14T06:41:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:41:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:41:37.969 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:37.967+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:41:37.969 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:37.968+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:41:37.970 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:37.968+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:41:37.970 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:37.968+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:41:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 28222464 bytes 2024-09-14T06:41:40.139 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:40.138+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:41:40.145 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:40.144+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:41:40.146 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:40.144+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:41:40.146 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:40.144+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:41:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 56908800 bytes 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2024-09-14T06:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:41:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:41:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:41:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:41:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:41:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:41:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:41:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:41:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:41:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 4466688 bytes 2024-09-14T06:41:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 26142720 bytes 2024-09-14T06:41:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 43663360 bytes 2024-09-14T06:41:46.362 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.358+0000 7ff305439640 -1 reset not still connected to 0x561ba5a149c0 2024-09-14T06:41:46.362 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.358+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:41:46.465 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.462+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:41:46.466 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.462+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:41:46.466 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.462+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:41:46.466 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.462+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:41:46.466 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.462+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:41:46.466 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.462+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:41:46.467 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:46.462+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:41:46.502 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:41:46.501+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:41:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 68086784 bytes 2024-09-14T06:41:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 78850048 bytes 2024-09-14T06:41:48.841 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:48.839+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:41:48.841 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:41:48.839+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:41:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 125026304 bytes 2024-09-14T06:41:52.012 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:41:52.010+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:41:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 169315328 bytes 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:41:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:41:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:41:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:41:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:41:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-14T06:41:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:41:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:41:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:41:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:41:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2024-09-14T06:41:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:41:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:41:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 10556416 bytes 2024-09-14T06:41:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 30800896 bytes 2024-09-14T06:41:55.426 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:55.425+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:41:55.426 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:55.425+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:41:55.428 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:55.427+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:41:55.429 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:41:55.427+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:41:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 76934144 bytes 2024-09-14T06:42:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 126659584 bytes 2024-09-14T06:42:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:42:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:42:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:42:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2024-09-14T06:42:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:42:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:42:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:42:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2024-09-14T06:42:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:42:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:42:03.111 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:03.109+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:42:03.119 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:03.117+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:42:03.119 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:03.117+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:42:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 20111360 bytes 2024-09-14T06:42:05.666 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:05.664+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:42:05.666 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:05.664+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:42:05.666 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:05.664+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:42:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 85902336 bytes 2024-09-14T06:42:08.818 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:42:08.816+0000 7ff0846f3640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-09-14T06:42:09.300 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:42:08.822+0000 7ff083ef2640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2024-09-14T06:42:09.301 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:08.828+0000 7f5669c93640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 11 2024-09-14T06:42:09.301 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:08.831+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-09-14T06:42:09.301 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:08.848+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2024-09-14T06:42:09.301 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:08.855+0000 7f566d49a640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-09-14T06:42:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 113796096 bytes 2024-09-14T06:42:09.790 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:09.788+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:42:09.794 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:09.792+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:42:09.794 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:09.792+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:42:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:42:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:42:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2024-09-14T06:42:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:42:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:42:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:42:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:42:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:42:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-14T06:42:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:42:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:42:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:42:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:42:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:42:13.834 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:13.832+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:42:13.836 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:13.834+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:42:13.836 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:13.834+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:42:13.836 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:13.834+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:42:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 49454080 bytes 2024-09-14T06:42:16.632 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:16.631+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:42:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 88258560 bytes 2024-09-14T06:42:19.408 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:19.405+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:42:19.411 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:19.409+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:42:19.411 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:19.409+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:42:19.411 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:19.409+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:42:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 96628736 bytes 2024-09-14T06:42:21.140 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:21.139+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:42:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:42:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:42:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:42:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:42:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:42:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2024-09-14T06:42:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:42:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:42:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:42:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:42:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2024-09-14T06:42:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:42:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:42:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:42:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:42:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:42:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-14T06:42:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:42:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:42:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:42:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:42:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:42:23.472 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.470+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b930 2024-09-14T06:42:23.472 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.471+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:42:23.472 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.471+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:42:23.472 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.471+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:42:23.472 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.471+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:42:23.473 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.471+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:42:23.473 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.471+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:42:23.473 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:23.471+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:42:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 2454528 bytes 2024-09-14T06:42:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 15114240 bytes 2024-09-14T06:42:25.513 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:25.510+0000 7f567c0c9640 -1 reset not still connected to 0x565528eb8dd0 2024-09-14T06:42:25.513 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:25.510+0000 7f567c0c9640 -1 reset not still connected to 0x565539a6a680 2024-09-14T06:42:26.010 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:26.009+0000 7f566d49a640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:42:26.017 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:42:26.016+0000 7f566cc99640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:42:26.746 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:26.745+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:42:27.737 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:27.736+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:42:27.740 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:27.739+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:42:27.740 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:27.739+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:42:27.741 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:27.739+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:42:27.741 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:27.739+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:42:27.741 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:27.739+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:42:28.133 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:28.131+0000 7ff305439640 -1 reset not still connected to 0x561ba5d61790 2024-09-14T06:42:28.133 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:28.131+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:42:28.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:28.131+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:42:28.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:28.131+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:42:28.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:28.131+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:42:28.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:28.131+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:42:28.134 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:28.131+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:42:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 71698432 bytes 2024-09-14T06:42:29.728 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:42:29.727+0000 7ff091b1f640 -1 reset not still connected to 0x55e998457450 2024-09-14T06:42:29.729 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:42:29.727+0000 7ff091b1f640 -1 reset not still connected to 0x55e998457ee0 2024-09-14T06:42:29.730 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:42:29.729+0000 7ff091b1f640 -1 reset not still connected to 0x55e999399040 2024-09-14T06:42:30.451 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:30.449+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:42:30.452 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:30.451+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:42:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 97753088 bytes 2024-09-14T06:42:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 108391424 bytes 2024-09-14T06:42:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 153231360 bytes 2024-09-14T06:42:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 153839616 bytes 2024-09-14T06:42:34.215 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:34.213+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:42:34.217 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:34.215+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:42:34.217 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:34.215+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:42:34.217 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:34.215+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:42:34.482 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:34.479+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:42:34.482 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:34.479+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:42:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:800 IOs, 162110464 bytes 2024-09-14T06:42:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:42:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2024-09-14T06:42:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:42:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:42:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:42:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:42:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:42:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 1976320 bytes 2024-09-14T06:42:37.320 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:37.317+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:42:37.321 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:37.317+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:42:37.323 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:37.320+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:42:37.323 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:37.320+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:42:37.323 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:37.320+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:42:37.327 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:37.325+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:42:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 26740736 bytes 2024-09-14T06:42:39.178 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:39.177+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390af70 2024-09-14T06:42:39.178 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:39.177+0000 7f642e14d640 -1 reset not still connected to 0x55b7e3f0f520 2024-09-14T06:42:39.178 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:39.177+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:42:39.178 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:39.177+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:42:39.179 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:39.177+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:42:39.179 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:39.177+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:42:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 78328832 bytes 2024-09-14T06:42:42.123 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.121+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:42:42.190 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.188+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:42:42.196 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.195+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:42:42.196 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.195+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:42:42.196 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.195+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:42:42.196 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.195+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:42:42.197 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.195+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:42:42.197 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.195+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:42:42.197 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:42.195+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:42:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 138362880 bytes 2024-09-14T06:42:46.312 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:46.309+0000 7ff305439640 -1 reset not still connected to 0x561ba4420dd0 2024-09-14T06:42:46.312 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:46.309+0000 7ff305439640 -1 reset not still connected to 0x561ba4421380 2024-09-14T06:42:46.312 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:46.309+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:42:46.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:46.313+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:42:46.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:46.313+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:42:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 154431488 bytes 2024-09-14T06:42:46.910 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:46.908+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:42:46.910 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:46.908+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 168896512 bytes 2024-09-14T06:42:48.589 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:48.587+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:42:48.592 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:48.589+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:42:48.592 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:48.589+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:42:48.592 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:48.589+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:42:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:42:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:42:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:42:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:42:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:42:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2024-09-14T06:42:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:42:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:42:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2024-09-14T06:42:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:42:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:42:50.994 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:50.993+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:42:50.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:50.995+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:42:50.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:50.995+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:42:50.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:50.995+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:42:50.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:50.995+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:42:50.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:50.995+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:42:51.039 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.038+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c45b0 2024-09-14T06:42:51.040 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.038+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:42:51.042 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.041+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:42:51.043 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.041+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:42:51.043 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.041+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:42:51.043 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.041+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:42:51.043 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.041+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:42:51.043 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:51.041+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:42:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 66065408 bytes 2024-09-14T06:42:53.077 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.075+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:42:53.077 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.075+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:42:53.077 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.075+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:42:53.077 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.075+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:42:53.077 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.075+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:42:53.077 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.075+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:42:53.077 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.075+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:42:53.649 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561ba5d61790 2024-09-14T06:42:53.649 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:42:53.649 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:42:53.649 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:42:53.649 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:42:53.649 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:42:53.650 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:42:53.650 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:42:53.650 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:42:53.647+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:42:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 98354176 bytes 2024-09-14T06:42:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:42:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:42:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-14T06:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:42:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:42:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:42:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:42:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:42:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2024-09-14T06:42:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:42:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:42:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2024-09-14T06:42:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:42:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:42:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 23986176 bytes 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b1e0 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b930 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:42:56.790 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:42:56.788+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:42:57.268 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:57.267+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:42:57.271 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:57.269+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:42:57.271 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:57.269+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:42:57.271 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:57.269+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:42:57.271 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:42:57.269+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:43:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 70277120 bytes 2024-09-14T06:43:00.787 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:00.785+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:43:00.787 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:00.785+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:43:00.787 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:00.785+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:43:00.787 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:00.785+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:43:00.787 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:00.785+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:43:02.665 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:02.663+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390af70 2024-09-14T06:43:02.665 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:02.663+0000 7f642e14d640 -1 reset not still connected to 0x55b7e3f0f520 2024-09-14T06:43:02.665 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:02.663+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:43:02.665 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:02.663+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:43:02.666 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:02.664+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:43:02.666 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:02.665+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:43:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 169964544 bytes 2024-09-14T06:43:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:43:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:43:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:43:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:43:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:43:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:43:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2024-09-14T06:43:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:43:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:43:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 49498112 bytes 2024-09-14T06:43:14.029 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:14.027+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b450 2024-09-14T06:43:14.029 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:14.027+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:43:14.030 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:14.029+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:43:14.030 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:14.029+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:43:14.032 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:14.031+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:43:14.032 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:14.031+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:43:14.032 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:14.031+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:43:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:43:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:43:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2024-09-14T06:43:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:43:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:43:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2024-09-14T06:43:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:43:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:43:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 109690880 bytes 2024-09-14T06:43:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 140599296 bytes 2024-09-14T06:43:23.983 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:23.981+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:43:23.984 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:23.981+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:43:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 141211648 bytes 2024-09-14T06:43:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 144746496 bytes 2024-09-14T06:43:25.688 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:25.686+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:43:25.689 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:25.686+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:43:25.689 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:25.686+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:43:25.689 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:25.686+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:43:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 185117696 bytes 2024-09-14T06:43:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 203894784 bytes 2024-09-14T06:43:28.839 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.835+0000 7ff305439640 -1 reset not still connected to 0x561ba4421380 2024-09-14T06:43:28.839 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.835+0000 7ff305439640 -1 reset not still connected to 0x561ba4421860 2024-09-14T06:43:28.843 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.839+0000 7ff305439640 -1 reset not still connected to 0x561ba497e270 2024-09-14T06:43:28.843 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.839+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:43:28.843 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.839+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:43:28.843 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.839+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:43:28.843 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.839+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:43:28.843 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.839+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561ba5d61790 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:43:28.844 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:28.841+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:43:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:43:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:43:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2024-09-14T06:43:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:43:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:43:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-14T06:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:43:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:43:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2024-09-14T06:43:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:43:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:43:31.341 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:31.339+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:43:31.346 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:31.343+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:43:31.346 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:31.343+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:43:31.346 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:31.345+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:43:31.346 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:31.345+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:43:31.347 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:31.346+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:43:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 64128000 bytes 2024-09-14T06:43:37.233 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:37.231+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b450 2024-09-14T06:43:37.233 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:37.231+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:43:37.233 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:37.231+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:43:37.234 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:37.232+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:43:37.234 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:37.232+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:43:37.234 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:37.232+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:43:37.234 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:37.232+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:43:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 111947776 bytes 2024-09-14T06:43:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 112551936 bytes 2024-09-14T06:43:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:43:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:43:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:43:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:43:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:43:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2024-09-14T06:43:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:43:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:43:40.062 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:40.060+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:43:40.062 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:40.061+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:43:40.063 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:40.061+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:43:40.063 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:40.061+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:43:40.063 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:40.061+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:43:40.063 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:40.061+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:43:41.209 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:41.207+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:43:41.209 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:41.207+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:43:41.209 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:41.207+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:43:41.209 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:41.207+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:43:41.210 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:41.207+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:43:41.210 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:41.207+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:43:41.210 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:41.207+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:43:41.872 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:41.869+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:43:41.873 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:41.870+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:43:41.873 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:41.871+0000 7ff305439640 -1 reset not still connected to 0x561baf5d36c0 2024-09-14T06:43:41.873 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:41.871+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:43:41.874 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:41.873+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:43:41.874 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:41.873+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:43:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 78611456 bytes 2024-09-14T06:43:43.554 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:43.552+0000 7f642e14d640 -1 reset not still connected to 0x55b7e17adad0 2024-09-14T06:43:43.554 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:43.552+0000 7f642e14d640 -1 reset not still connected to 0x55b7e3f0f520 2024-09-14T06:43:43.556 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:43.554+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:43:43.557 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:43.554+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:43:43.557 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:43.556+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:43:43.560 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:43.557+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:43:43.809 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:43:43.807+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:43:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 163511296 bytes 2024-09-14T06:43:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:43:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:43:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:43:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:43:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:43:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2024-09-14T06:43:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:43:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2024-09-14T06:43:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:43:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:43:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:43:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:43:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:43:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-14T06:43:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:43:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:43:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:43:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:43:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=14 2024-09-14T06:43:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:43:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:43:48.480 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.478+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:43:48.480 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.478+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:43:48.820 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.819+0000 7ff305439640 -1 reset not still connected to 0x561ba5a149c0 2024-09-14T06:43:48.820 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.819+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:43:48.820 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.819+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:43:48.821 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.819+0000 7ff305439640 -1 reset not still connected to 0x561baf5d36c0 2024-09-14T06:43:48.822 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.821+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:43:48.822 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.821+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:43:48.823 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.821+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:43:48.823 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:48.821+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:43:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 30705664 bytes 2024-09-14T06:43:50.269 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:50.267+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:43:50.953 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:50.951+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:43:50.953 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:50.951+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:43:50.956 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:50.954+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:43:50.956 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:50.954+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:43:50.956 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:50.954+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:43:50.956 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:50.954+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:43:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 48899072 bytes 2024-09-14T06:43:52.001 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:51.999+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:43:52.002 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:52.001+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:43:52.006 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:52.001+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:43:52.006 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:52.002+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:43:52.006 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:52.002+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:43:52.007 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:52.006+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:43:52.023 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:52.021+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:43:52.030 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:52.023+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:43:52.030 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:52.025+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:43:52.031 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:52.025+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:43:52.031 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:52.025+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:43:52.031 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:52.025+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:43:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 94383104 bytes 2024-09-14T06:43:53.126 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.125+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:43:53.127 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.125+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:43:53.127 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.125+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:43:53.129 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.127+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab5f0 2024-09-14T06:43:53.130 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.129+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:43:53.130 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.129+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:43:53.130 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.129+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:43:53.131 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.129+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:43:53.131 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.129+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:43:53.131 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.129+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:43:53.131 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:43:53.130+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:43:56.382 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:56.378+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:43:56.382 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:56.378+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:43:56.385 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:56.384+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:43:56.388 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:56.387+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:43:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 138322944 bytes 2024-09-14T06:43:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 175392768 bytes 2024-09-14T06:43:59.313 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba497e270 2024-09-14T06:43:59.313 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:43:59.313 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba5d61790 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6b6c0 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:43:59.314 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.311+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:43:59.921 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:43:59.920+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:44:00.013 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:00.010+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:44:00.013 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:00.012+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:44:00.014 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:00.013+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:44:00.014 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:00.013+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:44:00.014 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:00.013+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:44:00.014 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:00.013+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:44:00.014 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:00.013+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:44:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 226126848 bytes 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:44:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:44:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2024-09-14T06:44:02.802 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:02.799+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:02.802 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:02.799+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:44:02.802 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:02.799+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:02.802 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:02.799+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:44:03.139 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:03.138+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:44:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:44:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:44:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:44:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:44:06.455 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:06.454+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:44:06.456 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:06.454+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:44:06.456 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:06.454+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:44:06.456 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:06.454+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:44:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 58076160 bytes 2024-09-14T06:44:07.568 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.564+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b930 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.564+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:44:07.570 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:07.566+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:44:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 68425728 bytes 2024-09-14T06:44:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 97638400 bytes 2024-09-14T06:44:10.411 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:10.409+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:44:10.411 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:10.410+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:44:10.411 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:10.410+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 121292800 bytes 2024-09-14T06:44:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 172572672 bytes 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:44:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:44:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:44:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:44:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:44:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2024-09-14T06:44:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:44:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:44:17.600 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c45b0 2024-09-14T06:44:17.600 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:44:17.600 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab5f0 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:44:17.601 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.598+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:44:17.835 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:17.833+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:44:17.835 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:17.833+0000 7ff305439640 -1 reset not still connected to 0x561baf5d36c0 2024-09-14T06:44:17.835 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:17.833+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:44:17.835 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:17.833+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:44:17.835 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:17.833+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:44:17.835 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:17.833+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:44:17.973 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.971+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:17.973 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:17.971+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:18.121 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.120+0000 7ff305439640 -1 reset not still connected to 0x561ba497e270 2024-09-14T06:44:18.125 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.123+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:44:18.125 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.123+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba5d61790 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:44:18.127 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.125+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6b6c0 2024-09-14T06:44:18.128 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.127+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:44:18.128 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.127+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:44:18.128 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.127+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:44:18.128 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.127+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:44:18.128 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.127+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:44:18.128 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:18.127+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:44:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 82515968 bytes 2024-09-14T06:44:21.099 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:21.096+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:21.099 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:21.098+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:44:21.102 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:21.101+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:44:21.102 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:21.101+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:21.102 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:21.101+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:44:21.102 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:21.101+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:44:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:44:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2024-09-14T06:44:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:44:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:44:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:44:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:44:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:44:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-14T06:44:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:44:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:44:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2024-09-14T06:44:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:44:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:44:23.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 30568448 bytes 2024-09-14T06:44:24.968 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.966+0000 7ff305439640 -1 reset not still connected to 0x561ba5a149c0 2024-09-14T06:44:24.968 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.966+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:44:24.969 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.968+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:44:24.971 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.970+0000 7ff305439640 -1 reset not still connected to 0x561baf5d36c0 2024-09-14T06:44:24.971 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.970+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:44:24.971 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.970+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:44:24.972 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.970+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:44:24.972 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:24.970+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:44:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 91879424 bytes 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:44:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:44:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:44:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2024-09-14T06:44:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:44:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:44:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:44:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:44:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:44:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-14T06:44:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:44:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:44:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2024-09-14T06:44:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:44:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:44:33.840 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.838+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:44:33.844 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.843+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:44:33.846 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.844+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:44:33.846 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.844+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:44:33.846 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.844+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:33.846 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.844+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:44:33.846 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.844+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:44:33.846 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.844+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:33.848 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.847+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:44:33.848 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.847+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:44:33.848 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.847+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:44:33.848 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.847+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:44:33.923 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:44:33.922+0000 7ff091b1f640 -1 reset not still connected to 0x55e9984572b0 2024-09-14T06:44:33.923 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:44:33.922+0000 7ff091b1f640 -1 reset not still connected to 0x55e99a025e10 2024-09-14T06:44:33.923 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:44:33.922+0000 7ff091b1f640 -1 reset not still connected to 0x55e99cbd1380 2024-09-14T06:44:33.923 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:44:33.922+0000 7ff091b1f640 -1 reset not still connected to 0x55e9b78a6000 2024-09-14T06:44:33.982 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.981+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:33.983 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.981+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:44:33.984 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.982+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:44:33.984 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.982+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:33.984 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.982+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:44:33.984 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.982+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:44:33.984 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.982+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:44:33.984 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:33.982+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:44:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 93220864 bytes 2024-09-14T06:44:36.689 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:36.686+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:44:36.943 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:36.940+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:44:36.943 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:36.940+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:44:36.943 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:36.940+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:44:36.944 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:36.940+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:44:36.944 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:36.940+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:44:37.996 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.993+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b1e0 2024-09-14T06:44:37.996 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.993+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b930 2024-09-14T06:44:37.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.993+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:44:37.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.993+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:44:37.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.994+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:44:37.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.994+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:37.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.994+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:44:37.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.994+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:44:37.997 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.994+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:37.998 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.995+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:44:37.999 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.996+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:44:37.999 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.996+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:44:37.999 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:37.996+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:44:38.825 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.822+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b1e0 2024-09-14T06:44:38.825 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.823+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b930 2024-09-14T06:44:38.825 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.823+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:44:38.830 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:44:38.830 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:44:38.830 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:44:38.830 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:44:38.830 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:44:38.830 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:44:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:44:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:44:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:44:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:38.828+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:44:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 168733696 bytes 2024-09-14T06:44:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:44:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:44:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:44:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:44:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2024-09-14T06:44:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:44:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:44:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=9 2024-09-14T06:44:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:44:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:44:43.535 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:43.531+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:44:43.536 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:44:43.533+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:44:44.585 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:44.583+0000 7f642e14d640 -1 reset not still connected to 0x55b7e3f0f520 2024-09-14T06:44:44.589 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:44.588+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:44:44.589 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:44.588+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:44:44.589 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:44.588+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:44:44.590 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:44.588+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:44:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 65545216 bytes 2024-09-14T06:44:46.881 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:46.880+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:44:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 91748352 bytes 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:44:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:44:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:44:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2024-09-14T06:44:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:44:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:44:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:44:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:44:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:44:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-14T06:44:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:44:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:44:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=7 2024-09-14T06:44:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:44:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:44:53.391 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:53.388+0000 7f642e14d640 -1 reset not still connected to 0x55b7e895bad0 2024-09-14T06:44:53.391 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:53.388+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:44:54.325 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:54.323+0000 7f642e14d640 -1 reset not still connected to 0x55b7e3f0f520 2024-09-14T06:44:54.329 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:54.326+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:44:54.332 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:54.330+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:44:54.333 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:54.330+0000 7f642e14d640 -1 reset not still connected to 0x55b7e895bad0 2024-09-14T06:44:54.333 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:54.330+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:44:54.333 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:54.330+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:44:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 82495488 bytes 2024-09-14T06:44:57.468 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:57.466+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:44:57.470 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:44:57.469+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:44:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:44:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:44:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2024-09-14T06:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:44:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:44:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:44:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-14T06:44:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:44:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:44:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:44:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:44:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2024-09-14T06:44:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:44:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:44:58.631 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:58.630+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:44:58.632 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:58.630+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:44:58.632 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:58.630+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:44:58.632 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:44:58.630+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:45:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 46572544 bytes 2024-09-14T06:45:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:45:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:45:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:45:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2024-09-14T06:45:03.849 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:45:03.848+0000 7f0c69033640 -1 reset not still connected to 0x562cf73ba340 2024-09-14T06:45:03.849 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:45:03.848+0000 7f0c69033640 -1 reset not still connected to 0x562cfd4fc1a0 2024-09-14T06:45:03.849 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:45:03.848+0000 7f0c69033640 -1 reset not still connected to 0x562d0f698ea0 2024-09-14T06:45:03.849 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:45:03.848+0000 7f0c69033640 -1 reset not still connected to 0x562d13e44d00 2024-09-14T06:45:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:45:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:45:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:45:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:45:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:45:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 623616 bytes 2024-09-14T06:45:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 18941952 bytes 2024-09-14T06:45:06.542 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:06.539+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:45:06.542 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:06.539+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:45:06.542 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:06.541+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:45:06.542 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:06.541+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:45:06.543 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:06.541+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:45:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 67785728 bytes 2024-09-14T06:45:12.907 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.905+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b380 2024-09-14T06:45:12.907 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.905+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b450 2024-09-14T06:45:12.908 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.905+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:45:12.908 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.905+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:45:12.908 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.905+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:45:12.908 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.905+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:45:12.908 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.907+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:45:12.909 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.907+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:45:12.909 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.907+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:45:12.909 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:45:12.907+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:45:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 149210112 bytes 2024-09-14T06:45:14.836 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:14.835+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:45:14.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:14.835+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:45:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:45:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:45:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2024-09-14T06:45:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:45:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:45:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:45:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2024-09-14T06:45:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:45:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:45:14.976 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:14.975+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:45:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 43284480 bytes 2024-09-14T06:45:17.028 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:17.026+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:45:17.034 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:17.033+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:45:17.034 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:17.033+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:45:17.034 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:17.033+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:45:17.036 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:17.035+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:45:17.036 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:17.035+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:45:17.775 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:17.773+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:45:17.775 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:17.773+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:45:17.776 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:17.773+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:45:17.776 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:17.773+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:45:17.776 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:17.775+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:45:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:45:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:45:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:45:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:45:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:45:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2024-09-14T06:45:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2024-09-14T06:45:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:45:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:45:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:45:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-14T06:45:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:45:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:45:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:45:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:45:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2024-09-14T06:45:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:45:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:45:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:45:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:45:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:45:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 47333376 bytes 2024-09-14T06:45:23.354 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:23.353+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:45:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 94839808 bytes 2024-09-14T06:45:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 153756672 bytes 2024-09-14T06:45:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 196426752 bytes 2024-09-14T06:45:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:45:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:45:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:45:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-14T06:45:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:45:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:45:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2024-09-14T06:45:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:45:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:45:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 68682752 bytes 2024-09-14T06:45:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 70696960 bytes 2024-09-14T06:45:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 99520512 bytes 2024-09-14T06:45:41.581 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.579+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:45:41.589 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.587+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:45:41.593 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.591+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:45:41.593 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.591+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:45:41.593 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.591+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:45:41.593 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.591+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:45:41.593 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.591+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:45:41.593 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:41.591+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:45:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 116251648 bytes 2024-09-14T06:45:42.255 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.254+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:45:42.256 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.254+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:45:42.259 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.258+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:45:42.259 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.258+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561bb12e76c0 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:45:42.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:42.265+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:45:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:45:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:45:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:45:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:45:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2024-09-14T06:45:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:45:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:45:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:45:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2024-09-14T06:45:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:45:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:45:44.278 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.277+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:45:44.280 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.277+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:45:44.282 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.278+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:45:44.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.278+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:45:44.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.278+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:45:44.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.278+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:45:44.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.278+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:45:44.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:44.278+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:45:46.449 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.448+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:45:46.450 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.448+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:45:46.450 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.448+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:45:46.451 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.448+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:45:46.451 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.448+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.448+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.449+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.449+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.449+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.449+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.449+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.449+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:45:46.452 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:46.449+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:45:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 57061376 bytes 2024-09-14T06:45:49.678 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:49.676+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:45:49.678 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:49.676+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:45:49.678 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:49.676+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:45:49.678 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:45:49.676+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:45:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 108521472 bytes 2024-09-14T06:45:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 118957056 bytes 2024-09-14T06:45:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 159543296 bytes 2024-09-14T06:45:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:45:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:45:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:45:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:45:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:45:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:45:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:45:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2024-09-14T06:45:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:45:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:45:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 1116160 bytes 2024-09-14T06:45:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 4683776 bytes 2024-09-14T06:45:59.434 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.432+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:45:59.437 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.435+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:45:59.439 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.438+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:45:59.439 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.438+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:45:59.439 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.438+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:45:59.439 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.438+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:45:59.439 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.438+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:45:59.439 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.438+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:45:59.439 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:45:59.438+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:46:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:46:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:46:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:46:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-14T06:46:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:46:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2024-09-14T06:46:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:46:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:46:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:46:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:46:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:46:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-14T06:46:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:46:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:46:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:46:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:46:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:46:03.417 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:03.410+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:46:03.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:03.411+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:46:03.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:03.413+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:46:03.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:03.413+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:46:03.419 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:03.413+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:46:03.419 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:03.416+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:46:04.209 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:04.208+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:46:04.213 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:04.212+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:46:04.214 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:04.212+0000 7ff305439640 -1 reset not still connected to 0x561ba6c59e10 2024-09-14T06:46:04.214 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:04.212+0000 7ff305439640 -1 reset not still connected to 0x561baf5d36c0 2024-09-14T06:46:04.214 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:04.212+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:46:04.214 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:04.213+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:46:04.214 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:04.213+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:46:05.005 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.003+0000 7ff305439640 -1 reset not still connected to 0x561ba497e270 2024-09-14T06:46:05.005 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.003+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:46:05.006 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:46:05.006 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:46:05.006 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:46:05.006 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:46:05.006 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba5d61790 2024-09-14T06:46:05.006 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:46:05.008 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:46:05.008 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:46:05.008 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.004+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:46:05.011 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.008+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:46:05.011 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.008+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:46:05.011 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.008+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:46:05.012 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.008+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:46:05.012 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.008+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:46:05.012 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:05.008+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:46:06.296 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:06.294+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:46:06.296 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:06.294+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:46:06.296 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:06.294+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:46:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 96956416 bytes 2024-09-14T06:46:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 168784896 bytes 2024-09-14T06:46:11.055 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:11.053+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:46:11.055 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:11.053+0000 7ff305439640 -1 reset not still connected to 0x561be1662d00 2024-09-14T06:46:11.723 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:46:11.718+0000 7f567c0c9640 -1 reset not still connected to 0x56553b126270 2024-09-14T06:46:11.724 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:46:11.718+0000 7f567c0c9640 -1 reset not still connected to 0x565549ed2f70 2024-09-14T06:46:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 195307520 bytes 2024-09-14T06:46:12.700 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.699+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:46:12.700 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.699+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:46:12.704 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.703+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:46:12.704 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.703+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:46:12.706 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.705+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:46:12.706 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.705+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:46:12.707 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.705+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:46:12.708 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.707+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:46:12.709 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.707+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:46:12.709 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.707+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:46:12.709 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.707+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:46:12.709 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.707+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:46:12.709 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:12.707+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:46:13.288 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:13.283+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:46:13.288 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:13.283+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:46:13.288 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:13.283+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:46:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:46:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:46:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:46:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2024-09-14T06:46:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:46:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:46:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:46:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:46:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:46:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-14T06:46:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:46:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:46:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2024-09-14T06:46:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:46:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:46:14.269 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:14.265+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:46:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 15467520 bytes 2024-09-14T06:46:15.149 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.147+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:46:15.151 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.150+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:46:15.152 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.151+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:46:15.153 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.151+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:46:15.153 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.151+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:46:15.153 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.151+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:46:15.153 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.151+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:46:15.154 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.153+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:46:15.154 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.153+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:46:15.569 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.567+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:46:15.571 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.570+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:46:15.572 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.570+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:46:15.575 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.574+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:46:15.577 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.576+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:46:15.577 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.576+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:46:15.669 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b450 2024-09-14T06:46:15.670 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:46:15.670 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:46:15.670 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:46:15.670 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:46:15.670 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:46:15.670 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:46:15.670 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:15.668+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:46:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 35958784 bytes 2024-09-14T06:46:16.537 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:46:16.536+0000 7ff091b1f640 -1 reset not still connected to 0x55e9974984e0 2024-09-14T06:46:16.538 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:46:16.536+0000 7ff091b1f640 -1 reset not still connected to 0x55e9974996c0 2024-09-14T06:46:16.547 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:46:16.545+0000 7ff091b1f640 -1 reset not still connected to 0x55e998456270 2024-09-14T06:46:16.547 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:46:16.545+0000 7ff091b1f640 -1 reset not still connected to 0x55e998457790 2024-09-14T06:46:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 66489344 bytes 2024-09-14T06:46:19.241 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.239+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:46:19.242 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.241+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:46:19.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7e895bad0 2024-09-14T06:46:19.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:46:19.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:46:19.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:46:19.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:46:19.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:19.244+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:46:19.435 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:46:19.434+0000 7f0c57c07640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 5 2024-09-14T06:46:19.436 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:46:19.435+0000 7f0c55c03640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 5 2024-09-14T06:46:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 121735168 bytes 2024-09-14T06:46:21.474 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:21.472+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:46:22.660 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:22.658+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:46:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 140148736 bytes 2024-09-14T06:46:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 140763136 bytes 2024-09-14T06:46:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 160871424 bytes 2024-09-14T06:46:25.547 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.546+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:46:25.549 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.546+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:46:25.558 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:46:25.559 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:25.557+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:46:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:46:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:46:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2024-09-14T06:46:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:46:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:46:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:46:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:46:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:46:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:46:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 626688 bytes 2024-09-14T06:46:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 16368640 bytes 2024-09-14T06:46:30.551 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:30.550+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:46:30.552 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:30.550+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:46:30.552 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:30.550+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:46:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 76590080 bytes 2024-09-14T06:46:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 123904000 bytes 2024-09-14T06:46:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:46:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:46:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:46:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:46:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2024-09-14T06:46:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:46:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2024-09-14T06:46:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:46:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:46:38.712 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:38.711+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:46:38.712 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:38.711+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:46:38.713 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:38.711+0000 7ff305439640 -1 reset not still connected to 0x561bcbfcba00 2024-09-14T06:46:38.722 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:38.718+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:46:38.722 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:38.718+0000 7ff305439640 -1 reset not still connected to 0x561be1662d00 2024-09-14T06:46:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 38744064 bytes 2024-09-14T06:46:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 39322624 bytes 2024-09-14T06:46:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 39958528 bytes 2024-09-14T06:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2024-09-14T06:46:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:46:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:46:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:46:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:46:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:46:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-14T06:46:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:46:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:46:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:46:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:46:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:46:45.265 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:46:45.263+0000 7f0c69033640 -1 reset not still connected to 0x562cf73ba340 2024-09-14T06:46:45.265 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:46:45.263+0000 7f0c69033640 -1 reset not still connected to 0x562d0f698ea0 2024-09-14T06:46:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 53825536 bytes 2024-09-14T06:46:49.393 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:49.391+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:46:49.398 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:49.395+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:46:49.398 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:49.395+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:46:49.399 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:49.396+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:46:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 77340672 bytes 2024-09-14T06:46:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 116685824 bytes 2024-09-14T06:46:53.020 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:46:53.017+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:46:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 134960128 bytes 2024-09-14T06:46:54.677 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:54.675+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:46:54.677 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:54.675+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:46:54.677 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:54.675+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:46:54.677 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:54.675+0000 7ff305439640 -1 reset not still connected to 0x561bcbfcba00 2024-09-14T06:46:54.677 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:54.675+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:46:54.678 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:46:54.675+0000 7ff305439640 -1 reset not still connected to 0x561be1662d00 2024-09-14T06:46:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 143535104 bytes 2024-09-14T06:46:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:46:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:46:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2024-09-14T06:46:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:46:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:46:57.872 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.870+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:46:57.873 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.872+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1229ba0 2024-09-14T06:46:57.873 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.872+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:46:57.873 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.872+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:46:57.882 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:46:57.884 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:46:57.881+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2024-09-14T06:47:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:47:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2024-09-14T06:47:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:47:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-14T06:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:47:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:47:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:47:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:47:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:47:03.813 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:03.811+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:47:03.817 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:03.816+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:47:03.817 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:03.816+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:47:03.817 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:03.816+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:47:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 77544448 bytes 2024-09-14T06:47:06.250 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:06.248+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:47:06.255 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:06.252+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed42e340 2024-09-14T06:47:06.255 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:06.252+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:47:06.255 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:06.252+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:47:06.255 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:06.252+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:47:06.255 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:06.252+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:47:06.255 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:06.252+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:47:09.866 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:47:09.863+0000 7f0c5940a640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2024-09-14T06:47:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 148227072 bytes 2024-09-14T06:47:10.354 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:10.352+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:47:10.355 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:10.352+0000 7ff305439640 -1 reset not still connected to 0x561bd539f2b0 2024-09-14T06:47:10.355 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:10.352+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e3860 2024-09-14T06:47:11.230 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:11.228+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:47:11.304 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:47:11.246+0000 7ff091b1f640 -1 reset not still connected to 0x55e998456b60 2024-09-14T06:47:11.304 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:47:11.251+0000 7ff091b1f640 -1 reset not still connected to 0x55e9984575f0 2024-09-14T06:47:11.304 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:11.228+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:47:11.305 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:11.240+0000 7ff305439640 -1 reset not still connected to 0x561ba497e1a0 2024-09-14T06:47:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:47:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:47:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:47:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:47:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2024-09-14T06:47:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:47:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:47:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:47:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:47:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=6 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:47:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:47:15.457 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:15.451+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:47:15.459 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:15.454+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:47:15.460 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:15.455+0000 7ff305439640 -1 reset not still connected to 0x561bd539f2b0 2024-09-14T06:47:15.460 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:15.455+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e3860 2024-09-14T06:47:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 35235840 bytes 2024-09-14T06:47:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 72208384 bytes 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:47:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:47:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:47:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:47:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2024-09-14T06:47:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:47:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=8 2024-09-14T06:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:47:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:47:19.704 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.702+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287930 2024-09-14T06:47:19.704 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.702+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:47:19.704 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.702+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:47:19.705 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:47:19.705 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:47:19.705 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53dee0 2024-09-14T06:47:19.705 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:47:19.705 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:47:19.706 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b808dfdd40 2024-09-14T06:47:19.706 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:47:19.706 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.703+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:19.707 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:19.706+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:47:20.415 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.413+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:47:20.415 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.414+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:47:20.417 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.416+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:47:20.418 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.416+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:47:20.418 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.416+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:47:20.420 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.419+0000 7ff305439640 -1 reset not still connected to 0x561bcbfcba00 2024-09-14T06:47:20.420 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.419+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:47:20.420 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:20.419+0000 7ff305439640 -1 reset not still connected to 0x561be1662d00 2024-09-14T06:47:20.834 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:20.831+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 46691328 bytes 2024-09-14T06:47:23.435 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:23.432+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:47:23.435 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:23.432+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:47:23.435 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:23.432+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:47:23.435 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:23.432+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:47:24.571 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:47:24.569+0000 7ff07d6e5640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2024-09-14T06:47:24.591 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:47:24.590+0000 7ff07f6e9640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2024-09-14T06:47:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 70672384 bytes 2024-09-14T06:47:25.828 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.827+0000 7ff305439640 -1 reset not still connected to 0x561ba497e410 2024-09-14T06:47:25.832 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba497e9c0 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba5a15860 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba5d60000 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba5d61790 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:47:25.833 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.831+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:47:25.834 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.832+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:47:25.834 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.832+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6b6c0 2024-09-14T06:47:25.834 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.832+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bb12e76c0 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bcbfca0d0 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bd539f2b0 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e3860 2024-09-14T06:47:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:25.835+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n journal ']' 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 128M ']' 2024-09-14T06:47:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=128M 2024-09-14T06:47:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:47:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' pool = image ']' 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ case "${pool}" in 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 1 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n '' ']' 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-14T06:47:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:47:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:47:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2024-09-14T06:47:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:47:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:47:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 608256 bytes 2024-09-14T06:47:29.359 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.355+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:47:29.359 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.355+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:47:29.359 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.358+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:47:29.359 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.358+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:47:29.363 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:29.364 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:29.359+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:30.358 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:30.357+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:47:30.360 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:30.357+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:47:30.360 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:30.357+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:47:30.360 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:30.357+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:47:30.360 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:30.357+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:47:30.360 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:30.358+0000 7ff305439640 -1 reset not still connected to 0x561bd539f2b0 2024-09-14T06:47:30.364 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:30.363+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:47:31.267 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:31.265+0000 7ff305439640 -1 reset not still connected to 0x561ba497e1a0 2024-09-14T06:47:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 60705792 bytes 2024-09-14T06:47:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 90622976 bytes 2024-09-14T06:47:35.273 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.271+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:47:35.274 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.271+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6add0 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561ba8f6b6c0 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bb12e7040 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bb12e76c0 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:47:35.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:47:35.276 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:47:35.276 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:47:35.276 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.273+0000 7ff305439640 -1 reset not still connected to 0x561bcbfca0d0 2024-09-14T06:47:35.276 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.274+0000 7ff305439640 -1 reset not still connected to 0x561bd539f2b0 2024-09-14T06:47:35.276 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.274+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e3860 2024-09-14T06:47:35.277 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:35.276+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:47:36.257 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:36.256+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:47:36.262 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:36.259+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:47:36.262 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:36.260+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:47:36.262 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:36.260+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:47:36.599 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.597+0000 7f567c0c9640 -1 reset not still connected to 0x5655409e85b0 2024-09-14T06:47:36.599 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.597+0000 7f567c0c9640 -1 reset not still connected to 0x565549ed2f70 2024-09-14T06:47:36.599 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.597+0000 7f567c0c9640 -1 reset not still connected to 0x56554a24e270 2024-09-14T06:47:36.599 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.597+0000 7f567c0c9640 -1 reset not still connected to 0x56554c521a00 2024-09-14T06:47:36.735 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.732+0000 7f567c0c9640 -1 reset not still connected to 0x565539e27ee0 2024-09-14T06:47:36.735 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.732+0000 7f567c0c9640 -1 reset not still connected to 0x56554889a8f0 2024-09-14T06:47:36.735 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.732+0000 7f567c0c9640 -1 reset not still connected to 0x565548cb80d0 2024-09-14T06:47:36.736 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.735+0000 7f567c0c9640 -1 reset not still connected to 0x56554a2f7930 2024-09-14T06:47:36.742 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:47:36.739+0000 7f567c0c9640 -1 reset not still connected to 0x56555485bba0 2024-09-14T06:47:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 107747328 bytes 2024-09-14T06:47:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:47:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:47:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:47:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_count=50 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local count 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_count=0 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2024-09-14T06:47:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2024-09-14T06:47:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2024-09-14T06:47:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-14T06:47:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'images: ' 2024-09-14T06:47:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 2 2024-09-14T06:47:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:47:37.352 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.351+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:47:37.356 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.352+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:47:37.356 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.354+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:47:37.357 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.356+0000 7ff305439640 -1 reset not still connected to 0x561ba64d4c30 2024-09-14T06:47:37.357 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.356+0000 7ff305439640 -1 reset not still connected to 0x561ba7769d40 2024-09-14T06:47:37.357 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.356+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:47:37.373 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.370+0000 7ff305439640 -1 reset not still connected to 0x561bb2866c30 2024-09-14T06:47:37.373 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.370+0000 7ff305439640 -1 reset not still connected to 0x561bb2867380 2024-09-14T06:47:37.373 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.370+0000 7ff305439640 -1 reset not still connected to 0x561bb287cea0 2024-09-14T06:47:37.373 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.370+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:47:37.373 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.370+0000 7ff305439640 -1 reset not still connected to 0x561bd539f2b0 2024-09-14T06:47:37.373 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.370+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e3860 2024-09-14T06:47:37.373 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:37.370+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed42e340 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:47:37.377 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:37.375+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ count=50 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 50 = 50 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2024-09-14T06:47:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2024-09-14T06:47:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2024-09-14T06:47:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-14T06:47:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2024-09-14T06:47:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2024-09-14T06:47:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:47:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=WARNING 2024-09-14T06:47:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test WARNING = ERROR 2024-09-14T06:47:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test WARNING = OK 2024-09-14T06:47:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2024-09-14T06:47:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 -ne 1 2024-09-14T06:47:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 30 2024-09-14T06:47:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.824+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:47:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.826+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:47:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.826+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:47:38.831 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.826+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:47:38.832 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.826+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1229ba0 2024-09-14T06:47:38.832 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.826+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:47:38.832 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.826+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:47:38.832 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.826+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:38.833 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:47:38.834 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:38.834 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:38.834 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:38.834 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.830+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:38.837 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:38.835+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:39.459 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.457+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b930 2024-09-14T06:47:39.459 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.457+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:47:39.459 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.457+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:47:39.459 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.457+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:47:39.459 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.457+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c4820 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.462+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.462+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12288f0 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.462+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1229ba0 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.462+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.462+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.462+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.462+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:47:39.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:39.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.463+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:39.495 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.493+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:39.496 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.493+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:39.496 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.493+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:47:39.496 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.493+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:39.496 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.493+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:39.498 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.497+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:39.499 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:39.497+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:40.137 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.134+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390af70 2024-09-14T06:47:40.137 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.134+0000 7f642e14d640 -1 reset not still connected to 0x55b7e3f0f520 2024-09-14T06:47:40.137 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.134+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:47:40.137 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.135+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:47:40.138 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.135+0000 7f642e14d640 -1 reset not still connected to 0x55b7e895bad0 2024-09-14T06:47:40.138 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.135+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed42e340 2024-09-14T06:47:40.138 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.137+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:47:40.138 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.137+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:47:40.138 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.137+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:47:40.144 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.142+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:47:40.144 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:40.142+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:47:41.231 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.230+0000 7f642e14d640 -1 reset not still connected to 0x55b7e2b21e10 2024-09-14T06:47:41.231 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.230+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b380 2024-09-14T06:47:41.232 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.230+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390b450 2024-09-14T06:47:41.232 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.230+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:47:41.232 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.230+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:47:41.232 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.230+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:47:41.242 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.237+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:47:41.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53dee0 2024-09-14T06:47:41.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:47:41.246 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.244+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:47:41.247 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.245+0000 7f642e14d640 -1 reset not still connected to 0x55b808dfdd40 2024-09-14T06:47:41.248 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.245+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:47:41.248 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.245+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:41.248 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:41.245+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:47:41.262 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.258+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:47:41.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:47:41.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:41.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:41.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:47:41.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:41.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:41.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:41.269 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:41.269 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:41.266+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:42.241 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.237+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:47:42.249 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.245+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:47:42.249 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.245+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:47:42.249 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.245+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:47:42.250 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.245+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:47:42.250 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.245+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:42.250 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.245+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:42.251 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.247+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:42.251 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.247+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:42.251 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.247+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:42.251 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.247+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:42.251 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.247+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:42.723 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.721+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9b930 2024-09-14T06:47:42.725 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.721+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:47:42.728 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.727+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:47:42.728 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.727+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:47:42.738 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c4820 2024-09-14T06:47:42.738 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:47:42.738 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12288f0 2024-09-14T06:47:42.738 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1229ba0 2024-09-14T06:47:42.738 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:47:42.739 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:47:42.739 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:47:42.739 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:47:42.739 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:47:42.739 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:47:42.739 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.735+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:47:42.740 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:47:42.740 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:42.741 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:42.741 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:47:42.741 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:42.741 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:42.741 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:42.741 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:42.741 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:42.738+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:43.338 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.336+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:47:43.345 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.344+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:47:43.348 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:43.349 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.347+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:43.853 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.851+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0f16820 2024-09-14T06:47:43.853 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.851+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c4340 2024-09-14T06:47:43.854 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.851+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c45b0 2024-09-14T06:47:43.854 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.851+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:47:43.854 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.851+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:47:43.854 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.851+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:47:43.854 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.851+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb6f31a00 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab5f0 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:47:43.858 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:47:43.859 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:47:43.859 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:47:43.859 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:43.856+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcf040b930 2024-09-14T06:47:44.048 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0232820 2024-09-14T06:47:44.049 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0233c70 2024-09-14T06:47:44.049 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0f16820 2024-09-14T06:47:44.049 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c4340 2024-09-14T06:47:44.049 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c45b0 2024-09-14T06:47:44.049 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:47:44.049 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:47:44.050 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:47:44.050 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb6f31a00 2024-09-14T06:47:44.050 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab5f0 2024-09-14T06:47:44.050 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:47:44.050 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:47:44.050 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:47:44.051 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:47:44.051 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:47:44.051 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:47:44.051 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:47:44.051 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:47:44.051 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:47:44.052 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:47:44.052 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:47:44.052 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:47:44.052 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcf040b930 2024-09-14T06:47:44.052 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0232820 2024-09-14T06:47:44.053 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:47:44.053 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.046+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:47:44.417 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.414+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:47:44.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.415+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:47:44.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.415+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:47:44.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.415+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:47:44.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.415+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:47:44.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.415+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:47:44.418 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.415+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:47:44.420 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.419+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:47:44.420 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.419+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:47:44.421 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.420+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:47:44.421 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:44.420+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcf040b930 2024-09-14T06:47:44.827 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:44.825+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:47:44.827 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:44.825+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:47:44.829 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:44.827+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:47:44.829 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:44.827+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:44.829 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:44.827+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:47:49.193 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.191+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a820 2024-09-14T06:47:49.196 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.195+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:47:49.198 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:47:49.198 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:47:49.198 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:47:49.198 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:47:49.199 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b808dfdd40 2024-09-14T06:47:49.199 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:47:49.199 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:49.199 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:49.196+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:47:49.274 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.273+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80dd0 2024-09-14T06:47:49.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.273+0000 7ff305439640 -1 reset not still connected to 0x561ba6b80ea0 2024-09-14T06:47:49.275 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.273+0000 7ff305439640 -1 reset not still connected to 0x561baa1e5860 2024-09-14T06:47:49.276 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.275+0000 7ff305439640 -1 reset not still connected to 0x561bb12e64e0 2024-09-14T06:47:49.279 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.277+0000 7ff305439640 -1 reset not still connected to 0x561bb2844dd0 2024-09-14T06:47:49.279 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.277+0000 7ff305439640 -1 reset not still connected to 0x561bc43d05b0 2024-09-14T06:47:49.280 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.279+0000 7ff305439640 -1 reset not still connected to 0x561bcbfcba00 2024-09-14T06:47:49.281 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.279+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e2750 2024-09-14T06:47:49.281 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:47:49.279+0000 7ff305439640 -1 reset not still connected to 0x561be1662d00 2024-09-14T06:47:51.070 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:51.065+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:47:51.071 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:51.066+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:47:51.071 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:51.067+0000 7f642e14d640 -1 reset not still connected to 0x55b808dfdd40 2024-09-14T06:47:51.072 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:51.067+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:47:51.072 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:51.067+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:51.072 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:51.068+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:47:51.809 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.808+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:47:51.811 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.809+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:47:51.811 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.809+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:47:51.811 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.809+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:47:51.812 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.809+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:47:51.812 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.809+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:47:51.812 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.809+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:47:51.812 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.809+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:47:51.812 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:51.810+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcf040b930 2024-09-14T06:47:53.100 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.095+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0232820 2024-09-14T06:47:53.101 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.096+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c4340 2024-09-14T06:47:53.103 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.098+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c45b0 2024-09-14T06:47:53.104 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:47:53.104 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:47:53.104 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:47:53.104 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb6f31a00 2024-09-14T06:47:53.104 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab5f0 2024-09-14T06:47:53.104 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:47:53.105 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:47:53.105 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.099+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:47:53.105 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.100+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:47:53.106 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:47:53.106 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:47:53.108 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:47:53.108 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:47:53.109 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:47:53.109 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:47:53.109 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:47:53.109 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.105+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcf040b930 2024-09-14T06:47:53.953 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.951+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:47:53.954 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.951+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:47:53.954 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.951+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:47:53.954 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:53.951+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b7e895bad0 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed42e340 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:47:54.519 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:54.514+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:47:55.400 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.397+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:47:55.403 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.399+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53dee0 2024-09-14T06:47:55.404 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.401+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:47:55.404 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.401+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:47:55.404 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.401+0000 7f642e14d640 -1 reset not still connected to 0x55b808dfdd40 2024-09-14T06:47:55.404 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.401+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:47:55.404 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.401+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:55.404 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.401+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:47:55.457 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4a0a9c0 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cf70 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53dee0 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b808dfdd40 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:47:55.458 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:47:55.455+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:47:57.575 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0f16820 2024-09-14T06:47:57.576 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c4340 2024-09-14T06:47:57.576 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c45b0 2024-09-14T06:47:57.576 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:47:57.576 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:47:57.576 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:47:57.576 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab5f0 2024-09-14T06:47:57.576 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.572+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:47:57.578 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:47:57.574+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcf040b930 2024-09-14T06:48:00.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:00.281+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:48:00.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:00.281+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:48:04.694 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:04.693+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:48:04.696 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:04.695+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:48:04.696 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:04.695+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:48:04.696 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:04.695+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:48:07.126 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.122+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cea0 2024-09-14T06:48:07.127 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.125+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53dee0 2024-09-14T06:48:07.127 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.125+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3344410 2024-09-14T06:48:07.127 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.125+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455a00 2024-09-14T06:48:07.127 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.125+0000 7f642e14d640 -1 reset not still connected to 0x55b808dfdd40 2024-09-14T06:48:07.134 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.132+0000 7f642e14d640 -1 reset not still connected to 0x55b80fd0c5b0 2024-09-14T06:48:07.134 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.133+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:48:07.134 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.133+0000 7f642e14d640 -1 reset not still connected to 0x55b818f25040 2024-09-14T06:48:07.505 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.504+0000 7f642e14d640 -1 reset not still connected to 0x55b81755a680 2024-09-14T06:48:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-14T06:48:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2024-09-14T06:48:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2024-09-14T06:48:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:48:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=OK 2024-09-14T06:48:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = ERROR 2024-09-14T06:48:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = OK 2024-09-14T06:48:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-14T06:48:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_count=50 2024-09-14T06:48:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local count 2024-09-14T06:48:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_count=0 2024-09-14T06:48:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2024-09-14T06:48:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2024-09-14T06:48:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2024-09-14T06:48:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T06:48:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'images: ' 2024-09-14T06:48:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 2 2024-09-14T06:48:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:48:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ count=50 2024-09-14T06:48:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 50 = 50 2024-09-14T06:48:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-14T06:48:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state 2024-09-14T06:48:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2024-09-14T06:48:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2024-09-14T06:48:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2024-09-14T06:48:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T06:48:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2024-09-14T06:48:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2024-09-14T06:48:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:48:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=OK 2024-09-14T06:48:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = ERROR 2024-09-14T06:48:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = OK 2024-09-14T06:48:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: compare many images' 2024-09-14T06:48:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.RQ7U70XNbJ/rbd-mirror.test.log 2024-09-14T06:48:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2024-09-14T06:48:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2024-09-14 06:48:07 TEST: compare many images 2024-09-14T06:48:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14 06:48:07 TEST: compare many images 2024-09-14T06:48:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:07.886 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.880+0000 7f642e14d640 -1 reset not still connected to 0x55b7e390af70 2024-09-14T06:48:07.887 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.880+0000 7f642e14d640 -1 reset not still connected to 0x55b7e3f0f520 2024-09-14T06:48:07.892 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.888+0000 7f642e14d640 -1 reset not still connected to 0x55b7e4287040 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b7e70ca5b0 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b7e895bad0 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed42e340 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b7ed53cc30 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b7f2a0e1a0 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3345790 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b7f3455c70 2024-09-14T06:48:07.893 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:48:07.889+0000 7f642e14d640 -1 reset not still connected to 0x55b80d0f0000 2024-09-14T06:48:08.815 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.813+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb0d9bee0 2024-09-14T06:48:08.815 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.813+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:48:08.815 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.813+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:48:08.815 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.813+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12288f0 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1229ba0 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:48:08.820 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:48:08.821 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:08.818+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:48:08.850 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:08.849+0000 7f566dc9b640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:48:08.852 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:08.851+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-09-14T06:48:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-14T06:48:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-14T06:48:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:12.419 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.417+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:48:12.419 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.417+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:48:12.421 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.420+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:48:12.424 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.423+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:48:12.424 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.423+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:48:12.424 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.423+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:48:12.424 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.423+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:48:12.426 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.425+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:48:12.427 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.425+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:48:12.427 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.425+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:48:12.427 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.425+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:48:12.430 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.429+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:48:12.430 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.429+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:48:12.430 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.429+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:48:12.454 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.452+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10de750 2024-09-14T06:48:12.454 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.452+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb10dfd40 2024-09-14T06:48:12.454 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.452+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb12280d0 2024-09-14T06:48:12.458 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.454+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb127cf70 2024-09-14T06:48:12.458 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.454+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb152e9c0 2024-09-14T06:48:12.460 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb33785b0 2024-09-14T06:48:12.460 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9faa000 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe334ea0 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335450 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901040 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbfd16dd0 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd62ef450 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae64e0 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706fad0 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd95d9d40 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce7f3f2b0 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce98968f0 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8270 2024-09-14T06:48:12.461 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:12.458+0000 7f14e8c4d640 -1 reset not still connected to 0x55bced2bb6c0 2024-09-14T06:48:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-14T06:48:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 5 closest matches: 2024-09-14T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_11 2024-09-14T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_13 2024-09-14T06:48:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_15 2024-09-14T06:48:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_17 2024-09-14T06:48:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_19 2024-09-14T06:48:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-14T06:48:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_1 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_1 2024-09-14T06:48:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_1 ']' 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-09-14T06:48:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-09-14T06:48:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 5 closest matches: 2024-09-14T06:48:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_11 2024-09-14T06:48:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_13 2024-09-14T06:48:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_15 2024-09-14T06:48:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_17 2024-09-14T06:48:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_19 2024-09-14T06:48:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-09-14T06:48:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-09-14T06:48:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:48:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-09-14T06:48:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:48:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-14T06:48:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-14T06:48:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.status 2024-09-14T06:48:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-14T06:48:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.status 2024-09-14T06:48:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:48:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:48:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13003], [object_number=14, tag_tid=1, entry_tid=13002]]], state=connected] 2024-09-14T06:48:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13003], [object_number=14, tag_tid=1, entry_tid=13002]]], state=connected] 2024-09-14T06:48:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.status 2024-09-14T06:48:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-14T06:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.status 2024-09-14T06:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-14T06:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.status 2024-09-14T06:48:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:48:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:48:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13003], [object_number=14, tag_tid=1, entry_tid=13002]]], state=connected] 2024-09-14T06:48:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13003], [object_number=14, tag_tid=1, entry_tid=13002]]], state=connected] 2024-09-14T06:48:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.status 2024-09-14T06:48:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_1': 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b08330bda 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b08330bda 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:39:34 2024 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:39:34 2024 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:39:34 2024 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b08330bda 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c4abde9d-746c-4375-8c7e-940acb9a2476 2024-09-14T06:48:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.export 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_1.export 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_1.export 2024-09-14T06:48:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.export 2024-09-14T06:48:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_1.export 2024-09-14T06:48:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_1.export 2024-09-14T06:48:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_1.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_1.export 2024-09-14T06:48:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:16.856 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:48:16.854+0000 7ff07feea640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 7 2024-09-14T06:48:16.858 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:48:16.857+0000 7ff0846f3640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 9 2024-09-14T06:48:16.858 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:48:16.857+0000 7ff07eee8640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 9 2024-09-14T06:48:19.037 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:19.035+0000 7f567c0c9640 -1 reset not still connected to 0x565539a6a680 2024-09-14T06:48:19.037 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:19.035+0000 7f567c0c9640 -1 reset not still connected to 0x56553b126270 2024-09-14T06:48:19.037 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:19.035+0000 7f567c0c9640 -1 reset not still connected to 0x5655409e84e0 2024-09-14T06:48:19.037 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:19.035+0000 7f567c0c9640 -1 reset not still connected to 0x5655409e85b0 2024-09-14T06:48:19.037 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:19.035+0000 7f567c0c9640 -1 reset not still connected to 0x565549ed2f70 2024-09-14T06:48:19.037 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:19.036+0000 7f567c0c9640 -1 reset not still connected to 0x56554a24e270 2024-09-14T06:48:19.038 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:19.036+0000 7f567c0c9640 -1 reset not still connected to 0x56554c521a00 2024-09-14T06:48:19.800 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:48:19.798+0000 7ff305439640 -1 reset not still connected to 0x561ba4421380 2024-09-14T06:48:19.805 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:48:19.804+0000 7ff305439640 -1 reset not still connected to 0x561ba5a14750 2024-09-14T06:48:19.805 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:48:19.804+0000 7ff305439640 -1 reset not still connected to 0x561ba5a155f0 2024-09-14T06:48:19.805 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:48:19.804+0000 7ff305439640 -1 reset not still connected to 0x561ba5b3dee0 2024-09-14T06:48:19.805 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:48:19.804+0000 7ff305439640 -1 reset not still connected to 0x561baf5d3860 2024-09-14T06:48:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-14T06:48:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-14T06:48:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-14T06:48:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_2 2024-09-14T06:48:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_2 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_2 ']' 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-09-14T06:48:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-09-14T06:48:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-14T06:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-14T06:48:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:48:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:48:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842]]], state=connected] 2024-09-14T06:48:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842], [object_number=17, tag_tid=1, entry_tid=14841], [object_number=16, tag_tid=1, entry_tid=14840]]], state=connected] 2024-09-14T06:48:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-14T06:48:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-14T06:48:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:48:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:48:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842]]], state=connected] 2024-09-14T06:48:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842], [object_number=17, tag_tid=1, entry_tid=14841], [object_number=16, tag_tid=1, entry_tid=14840]]], state=connected] 2024-09-14T06:48:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=14843' 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=14843' 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=14843' '!=' '' 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=14843' = '' 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=19, tag_tid=1, entry_tid=14843' 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:48:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=14843' 2024-09-14T06:48:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:48:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:48:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:48:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:48:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=14843' 2024-09-14T06:48:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:48:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=14843 2024-09-14T06:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 14843 2024-09-14T06:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_2 2024-09-14T06:48:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_2 ']' 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-09-14T06:48:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-09-14T06:48:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-14T06:48:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-14T06:48:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:48:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:48:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842]]], state=connected] 2024-09-14T06:48:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842]]], state=connected] 2024-09-14T06:48:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-09-14T06:48:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-14T06:48:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:48:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:48:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842]]], state=connected] 2024-09-14T06:48:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14843], [object_number=18, tag_tid=1, entry_tid=14842]]], state=connected] 2024-09-14T06:48:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.status 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_2': 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abdd261aef 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abdd261aef 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:39:46 2024 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:39:46 2024 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:39:46 2024 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abdd261aef 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4d55c6e0-6d39-4e17-953e-b541bdbdd40c 2024-09-14T06:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-14T06:48:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:48:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.export 2024-09-14T06:48:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_2.export 2024-09-14T06:48:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_2.export 2024-09-14T06:48:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.export 2024-09-14T06:48:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_2.export 2024-09-14T06:48:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:23.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_2.export 2024-09-14T06:48:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_2.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_2.export 2024-09-14T06:48:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2024-09-14T06:48:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-14T06:48:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:23.868 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:23.866+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-09-14T06:48:23.868 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:23.867+0000 7f566cc99640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-14T06:48:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_3 2024-09-14T06:48:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_3 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_3 ']' 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-09-14T06:48:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-09-14T06:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-09-14T06:48:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-09-14T06:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_3.status 2024-09-14T06:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.status 2024-09-14T06:48:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-09-14T06:48:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.status 2024-09-14T06:48:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:48:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:48:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9143], [object_number=10, tag_tid=1, entry_tid=9142]]], state=connected] 2024-09-14T06:48:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9143], [object_number=10, tag_tid=1, entry_tid=9142]]], state=connected] 2024-09-14T06:48:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.status 2024-09-14T06:48:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-09-14T06:48:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_3.status 2024-09-14T06:48:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.status 2024-09-14T06:48:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-09-14T06:48:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.status 2024-09-14T06:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:48:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:48:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9143], [object_number=10, tag_tid=1, entry_tid=9142]]], state=connected] 2024-09-14T06:48:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9143], [object_number=10, tag_tid=1, entry_tid=9142]]], state=connected] 2024-09-14T06:48:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.status 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_3': 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abe52d3671 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abe52d3671 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:39:58 2024 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:39:58 2024 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:39:58 2024 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abe52d3671 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 1adda67c-553e-480f-aaab-5f91942def23 2024-09-14T06:48:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.export 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_3.export 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_3.export 2024-09-14T06:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.export 2024-09-14T06:48:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_3.export 2024-09-14T06:48:28.772 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:28.769+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:48:28.772 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:28.770+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:48:28.774 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:28.772+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:48:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_3.export 2024-09-14T06:48:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_3.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_3.export 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:29.880 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:48:29.879+0000 7ff07e6e7640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2024-09-14T06:48:29.883 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:48:29.882+0000 7ff0846f3640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-14T06:48:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-14T06:48:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 5 closest matches: 2024-09-14T06:48:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_41 2024-09-14T06:48:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_43 2024-09-14T06:48:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_45 2024-09-14T06:48:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_47 2024-09-14T06:48:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status mirror/image_49 2024-09-14T06:48:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-14T06:48:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:48:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:48:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-14T06:48:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-14T06:48:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_4 2024-09-14T06:48:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_4 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_4 ']' 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-09-14T06:48:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-09-14T06:48:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-09-14T06:48:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:no valid command found; 5 closest matches: 2024-09-14T06:48:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_41 2024-09-14T06:48:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_43 2024-09-14T06:48:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_45 2024-09-14T06:48:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_47 2024-09-14T06:48:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush mirror/image_49 2024-09-14T06:48:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:48:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:48:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-09-14T06:48:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_4.status 2024-09-14T06:48:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.status 2024-09-14T06:48:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-09-14T06:48:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.status 2024-09-14T06:48:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:48:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:48:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9119], [object_number=10, tag_tid=1, entry_tid=9118]]], state=connected] 2024-09-14T06:48:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9119], [object_number=10, tag_tid=1, entry_tid=9118]]], state=connected] 2024-09-14T06:48:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.status 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:48:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-09-14T06:48:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_4.status 2024-09-14T06:48:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.status 2024-09-14T06:48:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-09-14T06:48:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.status 2024-09-14T06:48:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:48:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:48:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9119], [object_number=10, tag_tid=1, entry_tid=9118]]], state=connected] 2024-09-14T06:48:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9119], [object_number=10, tag_tid=1, entry_tid=9118]]], state=connected] 2024-09-14T06:48:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.status 2024-09-14T06:48:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-14T06:48:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-14T06:48:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_4': 2024-09-14T06:48:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b06844591 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b06844591 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:40:08 2024 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:40:08 2024 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:40:08 2024 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b06844591 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f17b5877-fb4d-419b-8143-db0310e0fe78 2024-09-14T06:48:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.export 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_4.export 2024-09-14T06:48:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_4.export 2024-09-14T06:48:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.export 2024-09-14T06:48:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_4.export 2024-09-14T06:48:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_4.export 2024-09-14T06:48:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_4.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_4.export 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:35.888 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:35.887+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-09-14T06:48:35.890 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:35.889+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2024-09-14T06:48:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-14T06:48:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-14T06:48:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-14T06:48:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-14T06:48:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-14T06:48:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-14T06:48:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_5 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_5 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_5 ']' 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-09-14T06:48:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-09-14T06:48:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-09-14T06:48:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-14T06:48:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-14T06:48:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-14T06:48:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:48:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:48:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=11181], [object_number=12, tag_tid=1, entry_tid=11180], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-14T06:48:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-14T06:48:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:48:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:48:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=11181], [object_number=12, tag_tid=1, entry_tid=11180], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=11181' 2024-09-14T06:48:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=11181' 2024-09-14T06:48:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=11181' '!=' '' 2024-09-14T06:48:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:48:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=11181' = '' 2024-09-14T06:48:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=13, tag_tid=1, entry_tid=11181' 2024-09-14T06:48:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:48:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:48:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=11181' 2024-09-14T06:48:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:48:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:48:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:48:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:48:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=11181' 2024-09-14T06:48:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:48:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:48:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=11181 2024-09-14T06:48:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 11181 2024-09-14T06:48:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_5 2024-09-14T06:48:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_5 ']' 2024-09-14T06:48:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-09-14T06:48:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-09-14T06:48:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-14T06:48:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-14T06:48:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:48:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:48:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-09-14T06:48:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-09-14T06:48:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:48:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-14T06:48:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-14T06:48:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-14T06:48:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:48:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:48:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11179], [object_number=14, tag_tid=1, entry_tid=11178]]], state=connected] 2024-09-14T06:48:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.status 2024-09-14T06:48:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_5': 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab396f5d7 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab396f5d7 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:40:16 2024 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:40:16 2024 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:40:16 2024 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab396f5d7 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 918a879e-0e79-4eeb-bda9-a09d88f2c701 2024-09-14T06:48:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.export 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_5.export 2024-09-14T06:48:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_5.export 2024-09-14T06:48:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.export 2024-09-14T06:48:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_5.export 2024-09-14T06:48:40.465 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:40.463+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:48:40.466 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:40.465+0000 7f14d6817640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:48:40.467 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:40.466+0000 7f14d7018640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:48:40.470 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:40.468+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:48:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_5.export 2024-09-14T06:48:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_5.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_5.export 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:42.753 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565523ee6f70 2024-09-14T06:48:42.754 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565539a6b520 2024-09-14T06:48:42.754 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565539e26680 2024-09-14T06:48:42.754 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565539e27ee0 2024-09-14T06:48:42.754 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x56554889aea0 2024-09-14T06:48:42.754 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565548cb80d0 2024-09-14T06:48:42.754 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x56554a2f7930 2024-09-14T06:48:42.755 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565523ee6f70 2024-09-14T06:48:42.755 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565539a6b520 2024-09-14T06:48:42.755 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565539e26680 2024-09-14T06:48:42.755 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565539e27ee0 2024-09-14T06:48:42.755 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x56554889aea0 2024-09-14T06:48:42.755 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x565548cb80d0 2024-09-14T06:48:42.755 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:42.749+0000 7f567c0c9640 -1 reset not still connected to 0x56554a2f7930 2024-09-14T06:48:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-14T06:48:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-14T06:48:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-14T06:48:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-14T06:48:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-14T06:48:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:48:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:48:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-14T06:48:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-14T06:48:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_6 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_6 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_6 ']' 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-09-14T06:48:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-09-14T06:48:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-14T06:48:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:48:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-09-14T06:48:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_6.status 2024-09-14T06:48:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.status 2024-09-14T06:48:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-09-14T06:48:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.status 2024-09-14T06:48:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:48:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:48:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14947], [object_number=18, tag_tid=1, entry_tid=14946]]], state=connected] 2024-09-14T06:48:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14947], [object_number=18, tag_tid=1, entry_tid=14946]]], state=connected] 2024-09-14T06:48:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.status 2024-09-14T06:48:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-09-14T06:48:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:48:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-09-14T06:48:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:48:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-09-14T06:48:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_6.status 2024-09-14T06:48:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.status 2024-09-14T06:48:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-09-14T06:48:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.status 2024-09-14T06:48:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:48:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:48:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14947], [object_number=18, tag_tid=1, entry_tid=14946]]], state=connected] 2024-09-14T06:48:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14947], [object_number=18, tag_tid=1, entry_tid=14946]]], state=connected] 2024-09-14T06:48:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.status 2024-09-14T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-14T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-14T06:48:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_6': 2024-09-14T06:48:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0d937ac93 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0d937ac93 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:40:28 2024 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:40:28 2024 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:41:29 2024 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0d937ac93 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e78048de-1ace-4fac-ab6d-da5d7fc92891 2024-09-14T06:48:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.export 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_6.export 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_6.export 2024-09-14T06:48:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.export 2024-09-14T06:48:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_6.export 2024-09-14T06:48:46.270 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.269+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:48:46.271 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.270+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:48:46.283 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.282+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:48:46.284 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.282+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:48:46.339 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.338+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:48:46.340 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.338+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:48:46.340 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.339+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:48:46.345 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:48:46.344+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:48:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_6.export 2024-09-14T06:48:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_6.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_6.export 2024-09-14T06:48:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2024-09-14T06:48:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-14T06:48:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:47.119 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:47.117+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2024-09-14T06:48:47.120 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:47.119+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 6 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-14T06:48:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-14T06:48:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_7 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_7 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_7 ']' 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:48:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-09-14T06:48:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_7.status 2024-09-14T06:48:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.status 2024-09-14T06:48:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-09-14T06:48:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.status 2024-09-14T06:48:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:48:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:48:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9543], [object_number=10, tag_tid=1, entry_tid=9542]]], state=connected] 2024-09-14T06:48:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9543], [object_number=10, tag_tid=1, entry_tid=9542]]], state=connected] 2024-09-14T06:48:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.status 2024-09-14T06:48:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_7.status 2024-09-14T06:48:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.status 2024-09-14T06:48:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-09-14T06:48:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.status 2024-09-14T06:48:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:48:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:48:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9543], [object_number=10, tag_tid=1, entry_tid=9542]]], state=connected] 2024-09-14T06:48:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9543], [object_number=10, tag_tid=1, entry_tid=9542]]], state=connected] 2024-09-14T06:48:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.status 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-14T06:48:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_7': 2024-09-14T06:48:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abcf5b4b1 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abcf5b4b1 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:40:42 2024 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:40:42 2024 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:40:42 2024 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abcf5b4b1 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 29cc3a03-ef81-4143-bb4c-3a6e6da4fd90 2024-09-14T06:48:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.export 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_7.export 2024-09-14T06:48:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_7.export 2024-09-14T06:48:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.export 2024-09-14T06:48:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_7.export 2024-09-14T06:48:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_7.export 2024-09-14T06:48:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_7.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_7.export 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-14T06:48:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:48:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:48:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:48:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-14T06:48:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-14T06:48:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:48:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-14T06:48:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:48:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:48:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-14T06:48:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:48:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:48:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:48:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_8 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:48:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:48:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:48:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:48:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_8 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_8 ']' 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:48:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:48:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:48:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:48:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-09-14T06:48:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:48:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:48:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:48:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:48:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:48:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:48:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-09-14T06:48:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-14T06:48:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:48:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:48:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:48:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:48:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:48:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-09-14T06:48:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-09-14T06:48:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:48:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-09-14T06:48:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:48:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:48:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-09-14T06:48:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_8.status 2024-09-14T06:48:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.status 2024-09-14T06:48:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-09-14T06:48:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.status 2024-09-14T06:48:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:48:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:48:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5802], [object_number=7, tag_tid=1, entry_tid=5799]]], state=connected] 2024-09-14T06:48:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5802], [object_number=7, tag_tid=1, entry_tid=5799]]], state=connected] 2024-09-14T06:48:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.status 2024-09-14T06:48:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:48:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_8.status 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.status 2024-09-14T06:48:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-09-14T06:48:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.status 2024-09-14T06:48:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:48:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:48:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:48:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5802], [object_number=7, tag_tid=1, entry_tid=5799]]], state=connected] 2024-09-14T06:48:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5802], [object_number=7, tag_tid=1, entry_tid=5799]]], state=connected] 2024-09-14T06:48:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.status 2024-09-14T06:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-14T06:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-14T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_8': 2024-09-14T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0def74e98 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0def74e98 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:40:50 2024 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:40:50 2024 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:41:51 2024 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0def74e98 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e374a1a5-506d-4393-8e35-0c68b3e6e1be 2024-09-14T06:48:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:48:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-14T06:48:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:48:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:48:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.export 2024-09-14T06:48:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_8.export 2024-09-14T06:48:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_8.export 2024-09-14T06:48:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.export 2024-09-14T06:48:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_8.export 2024-09-14T06:48:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:48:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_8.export 2024-09-14T06:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_8.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_8.export 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:48:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:48:57.135 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:57.134+0000 7f5668490640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-09-14T06:48:57.141 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:57.140+0000 7f5669c93640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-09-14T06:48:59.252 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:48:59.251+0000 7f567c0c9640 -1 reset not still connected to 0x56554c521a00 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:48:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-14T06:49:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_9 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_9 2024-09-14T06:49:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_9 ']' 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-09-14T06:49:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-09-14T06:49:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_9.status 2024-09-14T06:49:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:00.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.status 2024-09-14T06:49:00.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-09-14T06:49:00.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.status 2024-09-14T06:49:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:49:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:49:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12907], [object_number=14, tag_tid=1, entry_tid=12906]]], state=connected] 2024-09-14T06:49:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12907], [object_number=14, tag_tid=1, entry_tid=12906]]], state=connected] 2024-09-14T06:49:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.status 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-09-14T06:49:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_9.status 2024-09-14T06:49:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.status 2024-09-14T06:49:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-09-14T06:49:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.status 2024-09-14T06:49:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:49:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:49:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12907], [object_number=14, tag_tid=1, entry_tid=12906]]], state=connected] 2024-09-14T06:49:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12907], [object_number=14, tag_tid=1, entry_tid=12906]]], state=connected] 2024-09-14T06:49:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.status 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_9': 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab3ee89e18 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab3ee89e18 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:40:57 2024 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:40:57 2024 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:41:58 2024 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab3ee89e18 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: b2f6675b-581a-465c-8059-e54224aec795 2024-09-14T06:49:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-14T06:49:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:49:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.export 2024-09-14T06:49:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_9.export 2024-09-14T06:49:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_9.export 2024-09-14T06:49:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.export 2024-09-14T06:49:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_9.export 2024-09-14T06:49:02.268 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:49:02.266+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:49:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_9.export 2024-09-14T06:49:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_9.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_9.export 2024-09-14T06:49:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2024-09-14T06:49:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-14T06:49:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:02.855 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:49:02.853+0000 7f0c56c05640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-09-14T06:49:02.861 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:49:02.860+0000 7f0c58408640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 8 2024-09-14T06:49:02.862 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:49:02.862+0000 7f0c5bc0f640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 8 2024-09-14T06:49:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-14T06:49:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-14T06:49:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-14T06:49:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:49:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-14T06:49:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:49:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:49:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-14T06:49:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_10 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_10 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_10 ']' 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-09-14T06:49:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-14T06:49:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-09-14T06:49:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-09-14T06:49:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-14T06:49:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-14T06:49:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-14T06:49:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7670], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=7670], [object_number=9, tag_tid=1, entry_tid=7669], [object_number=8, tag_tid=1, entry_tid=7668], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-09-14T06:49:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-14T06:49:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-14T06:49:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7670], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=10, tag_tid=1, entry_tid=7670], [object_number=9, tag_tid=1, entry_tid=7669], [object_number=8, tag_tid=1, entry_tid=7668], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=10, tag_tid=1, entry_tid=7670' 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=10, tag_tid=1, entry_tid=7670' 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=7670' '!=' '' 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=10, tag_tid=1, entry_tid=7670' = '' 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=10, tag_tid=1, entry_tid=7670' 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:49:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=7670' 2024-09-14T06:49:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:49:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:49:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=10, tag_tid=1, entry_tid=7670' 2024-09-14T06:49:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=7670 2024-09-14T06:49:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 7670 2024-09-14T06:49:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_10 2024-09-14T06:49:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_10 ']' 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-09-14T06:49:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-14T06:49:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-09-14T06:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-09-14T06:49:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-09-14T06:49:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-14T06:49:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-14T06:49:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7670], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7670], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-09-14T06:49:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-09-14T06:49:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-14T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-14T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-14T06:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7670], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7670], [object_number=11, tag_tid=1, entry_tid=7667]]], state=connected] 2024-09-14T06:49:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.status 2024-09-14T06:49:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-14T06:49:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-14T06:49:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_10': 2024-09-14T06:49:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0fcc1040e 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0fcc1040e 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:41:09 2024 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:41:09 2024 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:41:09 2024 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0fcc1040e 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: cff6a01c-483d-4882-b3fa-88d7151d117b 2024-09-14T06:49:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.export 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_10.export 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_10.export 2024-09-14T06:49:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.export 2024-09-14T06:49:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_10.export 2024-09-14T06:49:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_10.export 2024-09-14T06:49:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_10.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_10.export 2024-09-14T06:49:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:08.866 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:49:08.865+0000 7f0c56c05640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-09-14T06:49:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-14T06:49:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-14T06:49:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-14T06:49:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-14T06:49:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_11 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_11 2024-09-14T06:49:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_11 ']' 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-09-14T06:49:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-14T06:49:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-14T06:49:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-14T06:49:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890]]], state=connected] 2024-09-14T06:49:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890], [object_number=9, tag_tid=1, entry_tid=7889], [object_number=8, tag_tid=1, entry_tid=7888]]], state=connected] 2024-09-14T06:49:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-09-14T06:49:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-09-14T06:49:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-14T06:49:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-14T06:49:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-14T06:49:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890]]], state=connected] 2024-09-14T06:49:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890], [object_number=9, tag_tid=1, entry_tid=7889], [object_number=8, tag_tid=1, entry_tid=7888]]], state=connected] 2024-09-14T06:49:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=11, tag_tid=1, entry_tid=7891' 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=11, tag_tid=1, entry_tid=7891' 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=7891' '!=' '' 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=11, tag_tid=1, entry_tid=7891' = '' 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=11, tag_tid=1, entry_tid=7891' 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:49:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=7891' 2024-09-14T06:49:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:49:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:49:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=11, tag_tid=1, entry_tid=7891' 2024-09-14T06:49:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=7891 2024-09-14T06:49:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 7891 2024-09-14T06:49:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_11 2024-09-14T06:49:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_11 ']' 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-09-14T06:49:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-09-14T06:49:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-09-14T06:49:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-14T06:49:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-14T06:49:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890]]], state=connected] 2024-09-14T06:49:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890]]], state=connected] 2024-09-14T06:49:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-09-14T06:49:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-14T06:49:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-14T06:49:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890]]], state=connected] 2024-09-14T06:49:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7891], [object_number=10, tag_tid=1, entry_tid=7890]]], state=connected] 2024-09-14T06:49:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.status 2024-09-14T06:49:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-14T06:49:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-14T06:49:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_11': 2024-09-14T06:49:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab4666f213 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab4666f213 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:41:16 2024 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:41:16 2024 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:41:16 2024 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab4666f213 2024-09-14T06:49:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: bd028ebe-e25b-46a9-90cf-0975dd83d626 2024-09-14T06:49:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-14T06:49:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:49:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.export 2024-09-14T06:49:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_11.export 2024-09-14T06:49:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_11.export 2024-09-14T06:49:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.export 2024-09-14T06:49:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_11.export 2024-09-14T06:49:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_11.export 2024-09-14T06:49:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_11.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_11.export 2024-09-14T06:49:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2024-09-14T06:49:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-14T06:49:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-14T06:49:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-14T06:49:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:49:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-14T06:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:49:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-14T06:49:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_12 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_12 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_12 ']' 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-09-14T06:49:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-14T06:49:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-09-14T06:49:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-14T06:49:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-14T06:49:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-14T06:49:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=7961], [object_number=8, tag_tid=1, entry_tid=7960], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-14T06:49:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-14T06:49:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=7961], [object_number=8, tag_tid=1, entry_tid=7960], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=7961' 2024-09-14T06:49:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=7961' 2024-09-14T06:49:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=7961' '!=' '' 2024-09-14T06:49:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:49:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=7961' = '' 2024-09-14T06:49:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=7961' 2024-09-14T06:49:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:49:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=7961' 2024-09-14T06:49:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:49:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:49:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=7961' 2024-09-14T06:49:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=7961 2024-09-14T06:49:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 7961 2024-09-14T06:49:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_12 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_12 ']' 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-09-14T06:49:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-14T06:49:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-09-14T06:49:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-09-14T06:49:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-09-14T06:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-14T06:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-14T06:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-14T06:49:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-14T06:49:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-14T06:49:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-14T06:49:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.status 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-14T06:49:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_12': 2024-09-14T06:49:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0e528d240 2024-09-14T06:49:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0e528d240 2024-09-14T06:49:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:41:25 2024 2024-09-14T06:49:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:41:25 2024 2024-09-14T06:49:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:41:25 2024 2024-09-14T06:49:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0e528d240 2024-09-14T06:49:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4d84b376-5a07-4140-b5f0-001d91f99f27 2024-09-14T06:49:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-14T06:49:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:49:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.export 2024-09-14T06:49:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_12.export 2024-09-14T06:49:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_12.export 2024-09-14T06:49:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.export 2024-09-14T06:49:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_12.export 2024-09-14T06:49:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_12.export 2024-09-14T06:49:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_12.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_12.export 2024-09-14T06:49:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2024-09-14T06:49:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-14T06:49:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:19.699 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:49:19.698+0000 7f0c5a40c640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-09-14T06:49:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-14T06:49:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-14T06:49:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-14T06:49:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-14T06:49:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-14T06:49:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_13 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:21.225 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:49:21.224+0000 7ff305439640 -1 reset not still connected to 0x561ba497e1a0 2024-09-14T06:49:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_13 2024-09-14T06:49:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_13 ']' 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-09-14T06:49:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-09-14T06:49:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-09-14T06:49:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-14T06:49:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-14T06:49:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-14T06:49:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-14T06:49:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:49:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:49:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:49:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=1 2024-09-14T06:49:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 1 2024-09-14T06:49:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_13 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_13 ']' 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-09-14T06:49:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-09-14T06:49:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-14T06:49:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-14T06:49:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-14T06:49:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-14T06:49:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6175], [object_number=6, tag_tid=1, entry_tid=6174]]], state=connected] 2024-09-14T06:49:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.status 2024-09-14T06:49:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-14T06:49:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_13': 2024-09-14T06:49:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab21523db3 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab21523db3 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:41:33 2024 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:41:33 2024 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:42:34 2024 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab21523db3 2024-09-14T06:49:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: fd8cbe76-1ddc-4166-ba80-5a2a09aab630 2024-09-14T06:49:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.export 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_13.export 2024-09-14T06:49:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_13.export 2024-09-14T06:49:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.export 2024-09-14T06:49:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_13.export 2024-09-14T06:49:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_13.export 2024-09-14T06:49:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_13.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_13.export 2024-09-14T06:49:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2024-09-14T06:49:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-14T06:49:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-14T06:49:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-14T06:49:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-14T06:49:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-14T06:49:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:49:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:49:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:49:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:49:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-14T06:49:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_14 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_14 2024-09-14T06:49:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_14 ']' 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-09-14T06:49:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-09-14T06:49:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-14T06:49:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-09-14T06:49:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-09-14T06:49:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-14T06:49:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-14T06:49:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-14T06:49:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-14T06:49:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-14T06:49:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-09-14T06:49:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:49:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-09-14T06:49:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:49:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:49:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=1 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 1 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_14 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_14 ']' 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-09-14T06:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-14T06:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-09-14T06:49:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-09-14T06:49:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-09-14T06:49:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-14T06:49:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-14T06:49:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-14T06:49:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-14T06:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-14T06:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-14T06:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:49:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:49:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4323], [object_number=6, tag_tid=1, entry_tid=4322]]], state=connected] 2024-09-14T06:49:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.status 2024-09-14T06:49:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:49:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-14T06:49:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_14': 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b01f796693 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b01f796693 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:41:40 2024 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:41:40 2024 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:42:41 2024 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b01f796693 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 7742eddb-d6d2-4551-8046-e286b5eb20fc 2024-09-14T06:49:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.export 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_14.export 2024-09-14T06:49:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_14.export 2024-09-14T06:49:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.export 2024-09-14T06:49:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_14.export 2024-09-14T06:49:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_14.export 2024-09-14T06:49:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_14.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_14.export 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-14T06:49:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-14T06:49:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-14T06:49:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-14T06:49:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-14T06:49:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_15 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_15 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_15 ']' 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-09-14T06:49:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2024-09-14T06:49:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-09-14T06:49:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_15.status 2024-09-14T06:49:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.status 2024-09-14T06:49:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-09-14T06:49:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.status 2024-09-14T06:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12451], [object_number=14, tag_tid=1, entry_tid=12450]]], state=connected] 2024-09-14T06:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12451], [object_number=14, tag_tid=1, entry_tid=12450]]], state=connected] 2024-09-14T06:49:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.status 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2024-09-14T06:49:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-09-14T06:49:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_15.status 2024-09-14T06:49:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.status 2024-09-14T06:49:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-09-14T06:49:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.status 2024-09-14T06:49:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:49:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:49:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12451], [object_number=14, tag_tid=1, entry_tid=12450]]], state=connected] 2024-09-14T06:49:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12451], [object_number=14, tag_tid=1, entry_tid=12450]]], state=connected] 2024-09-14T06:49:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.status 2024-09-14T06:49:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-14T06:49:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_15': 2024-09-14T06:49:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab6b2ac6c7 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab6b2ac6c7 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:41:44 2024 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:41:44 2024 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:42:45 2024 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab6b2ac6c7 2024-09-14T06:49:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: c1a0c853-7f46-4932-a716-fe7a679a3291 2024-09-14T06:49:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.export 2024-09-14T06:49:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_15.export 2024-09-14T06:49:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_15.export 2024-09-14T06:49:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.export 2024-09-14T06:49:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_15.export 2024-09-14T06:49:32.810 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:49:32.808+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:49:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_15.export 2024-09-14T06:49:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_15.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_15.export 2024-09-14T06:49:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2024-09-14T06:49:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-14T06:49:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:33.911 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:49:33.910+0000 7f0c55402640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 5 2024-09-14T06:49:33.914 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:49:33.913+0000 7f0c5ac0d640 -1 bdev(0x562cf0bb8c00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 7 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-14T06:49:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-14T06:49:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-14T06:49:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-14T06:49:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:49:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-14T06:49:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:49:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:49:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:49:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-14T06:49:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_16 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_16 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_16 ']' 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-09-14T06:49:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-14T06:49:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-09-14T06:49:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-09-14T06:49:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_16.status 2024-09-14T06:49:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.status 2024-09-14T06:49:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-09-14T06:49:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.status 2024-09-14T06:49:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7567], [object_number=10, tag_tid=1, entry_tid=7566]]], state=connected] 2024-09-14T06:49:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7567], [object_number=10, tag_tid=1, entry_tid=7566]]], state=connected] 2024-09-14T06:49:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.status 2024-09-14T06:49:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-09-14T06:49:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2024-09-14T06:49:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-09-14T06:49:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2024-09-14T06:49:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-09-14T06:49:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_16.status 2024-09-14T06:49:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.status 2024-09-14T06:49:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-09-14T06:49:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.status 2024-09-14T06:49:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7567], [object_number=10, tag_tid=1, entry_tid=7566]]], state=connected] 2024-09-14T06:49:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7567], [object_number=10, tag_tid=1, entry_tid=7566]]], state=connected] 2024-09-14T06:49:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.status 2024-09-14T06:49:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-14T06:49:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_16': 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b04299baf8 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b04299baf8 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:41:56 2024 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:41:56 2024 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:41:56 2024 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b04299baf8 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 024264fc-4375-40de-b9b5-21efe9dd171b 2024-09-14T06:49:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-14T06:49:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:49:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.export 2024-09-14T06:49:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_16.export 2024-09-14T06:49:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_16.export 2024-09-14T06:49:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.export 2024-09-14T06:49:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_16.export 2024-09-14T06:49:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_16.export 2024-09-14T06:49:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_16.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_16.export 2024-09-14T06:49:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2024-09-14T06:49:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-14T06:49:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-14T06:49:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-14T06:49:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-14T06:49:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-14T06:49:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_17 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_17 2024-09-14T06:49:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_17 ']' 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-09-14T06:49:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-09-14T06:49:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-14T06:49:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-14T06:49:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9349], [object_number=8, tag_tid=1, entry_tid=9348], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-09-14T06:49:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-09-14T06:49:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-14T06:49:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-14T06:49:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-14T06:49:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9349], [object_number=8, tag_tid=1, entry_tid=9348], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9349' 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9349' 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9349' '!=' '' 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9349' = '' 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=9349' 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:49:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9349' 2024-09-14T06:49:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:49:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:49:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:49:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9349' 2024-09-14T06:49:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:49:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:49:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=9349 2024-09-14T06:49:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 9349 2024-09-14T06:49:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_17 2024-09-14T06:49:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_17 ']' 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-09-14T06:49:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-09-14T06:49:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-14T06:49:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-14T06:49:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-14T06:49:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9347], [object_number=10, tag_tid=1, entry_tid=9346]]], state=connected] 2024-09-14T06:49:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.status 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-14T06:49:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_17': 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abe514250c 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abe514250c 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:42:03 2024 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:42:03 2024 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:42:03 2024 2024-09-14T06:49:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abe514250c 2024-09-14T06:49:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4ae0f354-d4f5-4040-b6a9-c2e601dea8cd 2024-09-14T06:49:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-14T06:49:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:49:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.export 2024-09-14T06:49:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_17.export 2024-09-14T06:49:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_17.export 2024-09-14T06:49:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.export 2024-09-14T06:49:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_17.export 2024-09-14T06:49:45.076 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:49:45.074+0000 7f6420d21640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:49:45.077 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:49:45.075+0000 7f641fd1f640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-14T06:49:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_17.export 2024-09-14T06:49:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_17.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_17.export 2024-09-14T06:49:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2024-09-14T06:49:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-14T06:49:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:46.176 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:49:46.174+0000 7ff091b1f640 -1 reset not still connected to 0x55e998457790 2024-09-14T06:49:46.251 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:49:46.250+0000 7ff305439640 -1 reset not still connected to 0x561ba497e1a0 2024-09-14T06:49:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-14T06:49:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-14T06:49:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-14T06:49:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-14T06:49:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:49:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-14T06:49:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:49:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-14T06:49:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_18 2024-09-14T06:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_18 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_18 ']' 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-09-14T06:49:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-09-14T06:49:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-09-14T06:49:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-14T06:49:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:49:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:49:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-09-14T06:49:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:49:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:49:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:49:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:49:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:49:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-09-14T06:49:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-09-14T06:49:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2024-09-14T06:49:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-09-14T06:49:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2024-09-14T06:49:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-09-14T06:49:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_18.status 2024-09-14T06:49:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.status 2024-09-14T06:49:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-09-14T06:49:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.status 2024-09-14T06:49:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8639], [object_number=10, tag_tid=1, entry_tid=8638]]], state=connected] 2024-09-14T06:49:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8639], [object_number=10, tag_tid=1, entry_tid=8638]]], state=connected] 2024-09-14T06:49:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.status 2024-09-14T06:49:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-09-14T06:49:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2024-09-14T06:49:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-09-14T06:49:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2024-09-14T06:49:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-09-14T06:49:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_18.status 2024-09-14T06:49:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.status 2024-09-14T06:49:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-09-14T06:49:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.status 2024-09-14T06:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8639], [object_number=10, tag_tid=1, entry_tid=8638]]], state=connected] 2024-09-14T06:49:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8639], [object_number=10, tag_tid=1, entry_tid=8638]]], state=connected] 2024-09-14T06:49:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.status 2024-09-14T06:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_18': 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b05458e3b4 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b05458e3b4 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:42:14 2024 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:42:14 2024 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:42:14 2024 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b05458e3b4 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4c74da77-fb8c-4edf-b616-d75cc8073f7a 2024-09-14T06:49:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.export 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_18.export 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_18.export 2024-09-14T06:49:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.export 2024-09-14T06:49:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_18.export 2024-09-14T06:49:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_18.export 2024-09-14T06:49:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_18.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_18.export 2024-09-14T06:49:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2024-09-14T06:49:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-14T06:49:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:51.942 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:49:51.940+0000 7f5669c93640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:49:51.943 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:49:51.942+0000 7f566c498640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:49:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-14T06:49:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-14T06:49:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-14T06:49:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-14T06:49:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-14T06:49:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:49:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:49:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:49:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:49:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_19 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_19 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_19 ']' 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-09-14T06:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:49:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:49:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:49:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-09-14T06:49:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_19.status 2024-09-14T06:49:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.status 2024-09-14T06:49:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-09-14T06:49:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.status 2024-09-14T06:49:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:49:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:49:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12319], [object_number=14, tag_tid=1, entry_tid=12318]]], state=connected] 2024-09-14T06:49:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12319], [object_number=14, tag_tid=1, entry_tid=12318]]], state=connected] 2024-09-14T06:49:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.status 2024-09-14T06:49:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-09-14T06:49:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_19.status 2024-09-14T06:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.status 2024-09-14T06:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-09-14T06:49:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.status 2024-09-14T06:49:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:49:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12319], [object_number=14, tag_tid=1, entry_tid=12318]]], state=connected] 2024-09-14T06:49:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12319], [object_number=14, tag_tid=1, entry_tid=12318]]], state=connected] 2024-09-14T06:49:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.status 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:49:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_19': 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab1f4d08a0 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab1f4d08a0 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:49:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:42:25 2024 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:42:25 2024 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:43:26 2024 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab1f4d08a0 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e79b7cdb-a05e-457a-a8e9-9cb0ebb47be2 2024-09-14T06:49:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:49:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.export 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_19.export 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_19.export 2024-09-14T06:49:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.export 2024-09-14T06:49:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_19.export 2024-09-14T06:49:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:49:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_19.export 2024-09-14T06:49:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_19.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_19.export 2024-09-14T06:49:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:49:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:49:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2024-09-14T06:49:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-14T06:49:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:49:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:49:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:49:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:49:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:49:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:49:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:49:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:49:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:49:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:49:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:49:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:49:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:49:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:49:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:49:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:49:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:49:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:49:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:49:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:49:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:49:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:49:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:49:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:49:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:49:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-14T06:50:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-14T06:50:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:50:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-14T06:50:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:50:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:50:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:50:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-14T06:50:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_20 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:50:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_20 2024-09-14T06:50:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:50:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_20 ']' 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-09-14T06:50:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-14T06:50:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:50:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:50:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-09-14T06:50:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-09-14T06:50:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2024-09-14T06:50:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-09-14T06:50:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2024-09-14T06:50:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-14T06:50:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-14T06:50:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.status 2024-09-14T06:50:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-14T06:50:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.status 2024-09-14T06:50:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12263], [object_number=14, tag_tid=1, entry_tid=12262]]], state=connected] 2024-09-14T06:50:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12263], [object_number=14, tag_tid=1, entry_tid=12262]]], state=connected] 2024-09-14T06:50:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.status 2024-09-14T06:50:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-09-14T06:50:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2024-09-14T06:50:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-09-14T06:50:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2024-09-14T06:50:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-14T06:50:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-14T06:50:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.status 2024-09-14T06:50:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-14T06:50:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.status 2024-09-14T06:50:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12263], [object_number=14, tag_tid=1, entry_tid=12262]]], state=connected] 2024-09-14T06:50:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12263], [object_number=14, tag_tid=1, entry_tid=12262]]], state=connected] 2024-09-14T06:50:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.status 2024-09-14T06:50:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:01.200 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:01.199+0000 7ff305439640 -1 reset not still connected to 0x561ba4420820 2024-09-14T06:50:01.226 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.225+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb11c45b0 2024-09-14T06:50:01.226 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.225+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb14ee9c0 2024-09-14T06:50:01.231 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.228+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860340 2024-09-14T06:50:01.231 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.228+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb1860680 2024-09-14T06:50:01.231 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.228+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb6f31a00 2024-09-14T06:50:01.231 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.228+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab5f0 2024-09-14T06:50:01.232 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.228+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fab930 2024-09-14T06:50:01.237 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.235+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcb9fabc70 2024-09-14T06:50:01.237 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.235+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe335a00 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.235+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcbe901c70 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.235+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd6ae76c0 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.235+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcd706ec30 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.236+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce58484e0 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.236+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897790 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.236+0000 7f14e8c4d640 -1 reset not still connected to 0x55bce9897860 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.236+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d8d00 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.236+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcec2d95f0 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.236+0000 7f14e8c4d640 -1 reset not still connected to 0x55bceee18b60 2024-09-14T06:50:01.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:01.236+0000 7f14e8c4d640 -1 reset not still connected to 0x55bcf040b930 2024-09-14T06:50:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_20': 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0b9726a94 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0b9726a94 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:42:38 2024 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:42:38 2024 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:43:39 2024 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0b9726a94 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 20518e8c-1242-4fea-815a-dc5ba3fff94f 2024-09-14T06:50:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.export 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_20.export 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_20.export 2024-09-14T06:50:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.export 2024-09-14T06:50:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_20.export 2024-09-14T06:50:02.083 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:50:02.082+0000 7f641b516640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-14T06:50:02.087 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:50:02.086+0000 7f641ed1d640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-09-14T06:50:02.087 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:50:02.086+0000 7f641cd19640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:50:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_20.export 2024-09-14T06:50:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_20.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_20.export 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-14T06:50:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-14T06:50:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_21 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_21 2024-09-14T06:50:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_21 ']' 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-09-14T06:50:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-14T06:50:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-14T06:50:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-14T06:50:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:50:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6486], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6486], [object_number=5, tag_tid=1, entry_tid=6485], [object_number=4, tag_tid=1, entry_tid=6484], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:50:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-14T06:50:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-14T06:50:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:50:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6486], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=6, tag_tid=1, entry_tid=6486], [object_number=5, tag_tid=1, entry_tid=6485], [object_number=4, tag_tid=1, entry_tid=6484], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=6, tag_tid=1, entry_tid=6486' 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=6, tag_tid=1, entry_tid=6486' 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6486' '!=' '' 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=6, tag_tid=1, entry_tid=6486' = '' 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=6, tag_tid=1, entry_tid=6486' 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:50:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:50:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=6486' 2024-09-14T06:50:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:50:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=6, tag_tid=1, entry_tid=6486' 2024-09-14T06:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=6486 2024-09-14T06:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 1 -a 1 -le 6486 2024-09-14T06:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_21 2024-09-14T06:50:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_21 ']' 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-14T06:50:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-14T06:50:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-14T06:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:50:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6486], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6486], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:50:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-09-14T06:50:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-09-14T06:50:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:50:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-14T06:50:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-14T06:50:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-14T06:50:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:50:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6486], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6486], [object_number=7, tag_tid=1, entry_tid=6483]]], state=connected] 2024-09-14T06:50:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.status 2024-09-14T06:50:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:06.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-14T06:50:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_21': 2024-09-14T06:50:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab8d491cd7 2024-09-14T06:50:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab8d491cd7 2024-09-14T06:50:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:42:51 2024 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:42:51 2024 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:43:52 2024 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab8d491cd7 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f6dd37cb-2a3d-4fff-b916-01cabd0100ec 2024-09-14T06:50:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-14T06:50:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:50:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.export 2024-09-14T06:50:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_21.export 2024-09-14T06:50:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_21.export 2024-09-14T06:50:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.export 2024-09-14T06:50:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_21.export 2024-09-14T06:50:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_21.export 2024-09-14T06:50:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_21.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_21.export 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:06.963 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:50:06.961+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:50:06.968 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:50:06.967+0000 7f566b496640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-14T06:50:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-14T06:50:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-14T06:50:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-14T06:50:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:50:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:50:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:50:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-14T06:50:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_22 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_22 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_22 ']' 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-09-14T06:50:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-14T06:50:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:50:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:50:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2024-09-14T06:50:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-09-14T06:50:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_22.status 2024-09-14T06:50:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.status 2024-09-14T06:50:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-09-14T06:50:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.status 2024-09-14T06:50:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13199], [object_number=14, tag_tid=1, entry_tid=13198]]], state=connected] 2024-09-14T06:50:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13199], [object_number=14, tag_tid=1, entry_tid=13198]]], state=connected] 2024-09-14T06:50:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.status 2024-09-14T06:50:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_22.status 2024-09-14T06:50:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.status 2024-09-14T06:50:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-09-14T06:50:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.status 2024-09-14T06:50:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13199], [object_number=14, tag_tid=1, entry_tid=13198]]], state=connected] 2024-09-14T06:50:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13199], [object_number=14, tag_tid=1, entry_tid=13198]]], state=connected] 2024-09-14T06:50:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.status 2024-09-14T06:50:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-14T06:50:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_22': 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0bae79d5b 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0bae79d5b 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:42:57 2024 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:42:57 2024 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:43:58 2024 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0bae79d5b 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: ce45c080-2e5a-4206-922a-9479ef07fcc3 2024-09-14T06:50:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-14T06:50:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:50:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.export 2024-09-14T06:50:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_22.export 2024-09-14T06:50:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_22.export 2024-09-14T06:50:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.export 2024-09-14T06:50:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_22.export 2024-09-14T06:50:12.126 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:12.124+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:50:12.126 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:12.125+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:50:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_22.export 2024-09-14T06:50:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_22.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_22.export 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-14T06:50:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-14T06:50:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-14T06:50:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-14T06:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_23 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_23 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_23 ']' 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-09-14T06:50:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_23.status 2024-09-14T06:50:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.status 2024-09-14T06:50:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-09-14T06:50:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.status 2024-09-14T06:50:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:50:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6735], [object_number=6, tag_tid=1, entry_tid=6734]]], state=connected] 2024-09-14T06:50:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6735], [object_number=6, tag_tid=1, entry_tid=6734]]], state=connected] 2024-09-14T06:50:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.status 2024-09-14T06:50:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_23.status 2024-09-14T06:50:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.status 2024-09-14T06:50:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-09-14T06:50:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.status 2024-09-14T06:50:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:50:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6735], [object_number=6, tag_tid=1, entry_tid=6734]]], state=connected] 2024-09-14T06:50:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6735], [object_number=6, tag_tid=1, entry_tid=6734]]], state=connected] 2024-09-14T06:50:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.status 2024-09-14T06:50:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-14T06:50:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-14T06:50:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_23': 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab872309fb 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab872309fb 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:43:09 2024 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:43:09 2024 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:43:09 2024 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab872309fb 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: de2410a2-37db-484f-bc2b-74965359a6b8 2024-09-14T06:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-14T06:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.export 2024-09-14T06:50:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_23.export 2024-09-14T06:50:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_23.export 2024-09-14T06:50:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.export 2024-09-14T06:50:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_23.export 2024-09-14T06:50:16.852 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.851+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:50:16.870 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.868+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:50:16.870 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.868+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:50:16.870 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.868+0000 7ff2f6009640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:50:16.870 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.869+0000 7ff2f680a640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:50:16.884 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.883+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:50:16.884 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.883+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:50:16.885 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:16.884+0000 7ff2f2802640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:50:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_23.export 2024-09-14T06:50:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_23.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_23.export 2024-09-14T06:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2024-09-14T06:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-14T06:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:17.990 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:50:17.989+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:50:17.990 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:50:17.989+0000 7f566d49a640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-09-14T06:50:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-14T06:50:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-14T06:50:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-14T06:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-14T06:50:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-14T06:50:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:50:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-14T06:50:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:50:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:50:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-14T06:50:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_24 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_24 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_24 ']' 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-09-14T06:50:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-09-14T06:50:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-14T06:50:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:50:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:50:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-09-14T06:50:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-09-14T06:50:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_24.status 2024-09-14T06:50:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.status 2024-09-14T06:50:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-09-14T06:50:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.status 2024-09-14T06:50:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13146], [object_number=15, tag_tid=1, entry_tid=13143]]], state=connected] 2024-09-14T06:50:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13146], [object_number=15, tag_tid=1, entry_tid=13143]]], state=connected] 2024-09-14T06:50:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.status 2024-09-14T06:50:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-09-14T06:50:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_24.status 2024-09-14T06:50:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.status 2024-09-14T06:50:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-09-14T06:50:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.status 2024-09-14T06:50:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13146], [object_number=15, tag_tid=1, entry_tid=13143]]], state=connected] 2024-09-14T06:50:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=13146], [object_number=15, tag_tid=1, entry_tid=13143]]], state=connected] 2024-09-14T06:50:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.status 2024-09-14T06:50:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-14T06:50:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_24': 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b03159ce08 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b03159ce08 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:43:17 2024 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:43:17 2024 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:44:18 2024 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b03159ce08 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 54006e0b-31cb-464e-8386-0d06a8a5ca7b 2024-09-14T06:50:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.export 2024-09-14T06:50:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_24.export 2024-09-14T06:50:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_24.export 2024-09-14T06:50:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.export 2024-09-14T06:50:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_24.export 2024-09-14T06:50:22.053 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:22.052+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:50:22.054 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:22.053+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:50:22.138 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:22.137+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:50:22.195 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:50:22.194+0000 7ff2f5007640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-14T06:50:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_24.export 2024-09-14T06:50:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_24.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_24.export 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:23.004 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:50:23.003+0000 7f566b496640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:50:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-14T06:50:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_25 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_25 2024-09-14T06:50:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_25 ']' 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2024-09-14T06:50:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-09-14T06:50:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_25.status 2024-09-14T06:50:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.status 2024-09-14T06:50:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-09-14T06:50:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.status 2024-09-14T06:50:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:50:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:50:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7591], [object_number=10, tag_tid=1, entry_tid=7590]]], state=connected] 2024-09-14T06:50:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7591], [object_number=10, tag_tid=1, entry_tid=7590]]], state=connected] 2024-09-14T06:50:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.status 2024-09-14T06:50:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-09-14T06:50:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_25.status 2024-09-14T06:50:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.status 2024-09-14T06:50:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-09-14T06:50:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.status 2024-09-14T06:50:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:50:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:50:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7591], [object_number=10, tag_tid=1, entry_tid=7590]]], state=connected] 2024-09-14T06:50:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7591], [object_number=10, tag_tid=1, entry_tid=7590]]], state=connected] 2024-09-14T06:50:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.status 2024-09-14T06:50:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-14T06:50:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_25': 2024-09-14T06:50:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abb7699cfa 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abb7699cfa 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:43:32 2024 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:43:32 2024 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:43:32 2024 2024-09-14T06:50:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abb7699cfa 2024-09-14T06:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 05088c9d-b732-4b8b-8866-585a38154473 2024-09-14T06:50:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.export 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_25.export 2024-09-14T06:50:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_25.export 2024-09-14T06:50:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.export 2024-09-14T06:50:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_25.export 2024-09-14T06:50:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_25.export 2024-09-14T06:50:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_25.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_25.export 2024-09-14T06:50:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2024-09-14T06:50:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-14T06:50:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-14T06:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-14T06:50:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-14T06:50:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-14T06:50:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:50:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:50:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-14T06:50:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-14T06:50:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_26 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_26 2024-09-14T06:50:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_26 ']' 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-09-14T06:50:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-14T06:50:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:50:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:50:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2024-09-14T06:50:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-09-14T06:50:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_26.status 2024-09-14T06:50:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.status 2024-09-14T06:50:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-09-14T06:50:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.status 2024-09-14T06:50:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:50:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10303], [object_number=10, tag_tid=1, entry_tid=10302]]], state=connected] 2024-09-14T06:50:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10303], [object_number=10, tag_tid=1, entry_tid=10302]]], state=connected] 2024-09-14T06:50:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.status 2024-09-14T06:50:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2024-09-14T06:50:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-09-14T06:50:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_26.status 2024-09-14T06:50:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.status 2024-09-14T06:50:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-09-14T06:50:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.status 2024-09-14T06:50:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:50:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10303], [object_number=10, tag_tid=1, entry_tid=10302]]], state=connected] 2024-09-14T06:50:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10303], [object_number=10, tag_tid=1, entry_tid=10302]]], state=connected] 2024-09-14T06:50:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.status 2024-09-14T06:50:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-14T06:50:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-14T06:50:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_26': 2024-09-14T06:50:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b04d5e34e9 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b04d5e34e9 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:43:41 2024 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:43:41 2024 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:44:42 2024 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b04d5e34e9 2024-09-14T06:50:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: d5a036b9-b0ad-4751-8b9e-0114c686f518 2024-09-14T06:50:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-14T06:50:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:50:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.export 2024-09-14T06:50:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_26.export 2024-09-14T06:50:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_26.export 2024-09-14T06:50:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.export 2024-09-14T06:50:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_26.export 2024-09-14T06:50:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_26.export 2024-09-14T06:50:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_26.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_26.export 2024-09-14T06:50:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-14T06:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T06:50:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T06:50:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:36.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:36.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_27 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:36.218 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:50:36.217+0000 7ff091b1f640 -1 reset not still connected to 0x55e998457790 2024-09-14T06:50:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_27 2024-09-14T06:50:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_27 ']' 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-14T06:50:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-14T06:50:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-14T06:50:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:50:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:50:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742]]], state=connected] 2024-09-14T06:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742], [object_number=17, tag_tid=1, entry_tid=14741], [object_number=16, tag_tid=1, entry_tid=14740]]], state=connected] 2024-09-14T06:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-14T06:50:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-14T06:50:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-14T06:50:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:50:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:50:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742]]], state=connected] 2024-09-14T06:50:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742], [object_number=17, tag_tid=1, entry_tid=14741], [object_number=16, tag_tid=1, entry_tid=14740]]], state=connected] 2024-09-14T06:50:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=19, tag_tid=1, entry_tid=14743' 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=19, tag_tid=1, entry_tid=14743' 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=14743' '!=' '' 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=19, tag_tid=1, entry_tid=14743' = '' 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=19, tag_tid=1, entry_tid=14743' 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=14743' 2024-09-14T06:50:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:50:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:50:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:50:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:50:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=19, tag_tid=1, entry_tid=14743' 2024-09-14T06:50:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=14743 2024-09-14T06:50:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 14743 2024-09-14T06:50:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_27 2024-09-14T06:50:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_27 ']' 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-09-14T06:50:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-09-14T06:50:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-14T06:50:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-14T06:50:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:50:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:50:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742]]], state=connected] 2024-09-14T06:50:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742]]], state=connected] 2024-09-14T06:50:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-14T06:50:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-14T06:50:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-14T06:50:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 4 2024-09-14T06:50:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 4 2024-09-14T06:50:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742]]], state=connected] 2024-09-14T06:50:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14743], [object_number=18, tag_tid=1, entry_tid=14742]]], state=connected] 2024-09-14T06:50:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.status 2024-09-14T06:50:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:50:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-14T06:50:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-14T06:50:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-14T06:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_27': 2024-09-14T06:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abf2968f31 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abf2968f31 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:43:50 2024 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:43:50 2024 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:44:51 2024 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abf2968f31 2024-09-14T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e5943704-039c-4472-9a4b-843eb8d348cb 2024-09-14T06:50:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.export 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_27.export 2024-09-14T06:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_27.export 2024-09-14T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.export 2024-09-14T06:50:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_27.export 2024-09-14T06:50:38.376 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:38.375+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:50:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_27.export 2024-09-14T06:50:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_27.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_27.export 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:40.100 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:50:40.099+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:50:40.108 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:50:40.107+0000 7f566dc9b640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:50:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-14T06:50:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-14T06:50:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-14T06:50:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-14T06:50:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-14T06:50:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:50:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:50:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:50:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-14T06:50:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-14T06:50:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_28 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_28 2024-09-14T06:50:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_28 ']' 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-09-14T06:50:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-09-14T06:50:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-14T06:50:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:50:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-09-14T06:50:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:50:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-09-14T06:50:42.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_28.status 2024-09-14T06:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.status 2024-09-14T06:50:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-09-14T06:50:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.status 2024-09-14T06:50:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11099], [object_number=14, tag_tid=1, entry_tid=11098]]], state=connected] 2024-09-14T06:50:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11099], [object_number=14, tag_tid=1, entry_tid=11098]]], state=connected] 2024-09-14T06:50:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.status 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2024-09-14T06:50:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-09-14T06:50:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_28.status 2024-09-14T06:50:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.status 2024-09-14T06:50:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-09-14T06:50:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.status 2024-09-14T06:50:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11099], [object_number=14, tag_tid=1, entry_tid=11098]]], state=connected] 2024-09-14T06:50:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11099], [object_number=14, tag_tid=1, entry_tid=11098]]], state=connected] 2024-09-14T06:50:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.status 2024-09-14T06:50:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_28': 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0fe563b67 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0fe563b67 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:44:06 2024 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:44:06 2024 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:45:07 2024 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0fe563b67 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 38c098c7-2e00-42a5-abec-4eee5b97cdd0 2024-09-14T06:50:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-14T06:50:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:50:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.export 2024-09-14T06:50:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_28.export 2024-09-14T06:50:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_28.export 2024-09-14T06:50:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.export 2024-09-14T06:50:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_28.export 2024-09-14T06:50:44.282 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:44.281+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:50:44.287 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:44.286+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:50:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_28.export 2024-09-14T06:50:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_28.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_28.export 2024-09-14T06:50:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2024-09-14T06:50:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-14T06:50:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-14T06:50:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-14T06:50:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:47.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-14T06:50:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_29 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_29 2024-09-14T06:50:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_29 ']' 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-09-14T06:50:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2024-09-14T06:50:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-14T06:50:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-14T06:50:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.status 2024-09-14T06:50:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-14T06:50:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.status 2024-09-14T06:50:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:50:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6379], [object_number=6, tag_tid=1, entry_tid=6378]]], state=connected] 2024-09-14T06:50:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6379], [object_number=6, tag_tid=1, entry_tid=6378]]], state=connected] 2024-09-14T06:50:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.status 2024-09-14T06:50:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-14T06:50:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.status 2024-09-14T06:50:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-14T06:50:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.status 2024-09-14T06:50:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6379], [object_number=6, tag_tid=1, entry_tid=6378]]], state=connected] 2024-09-14T06:50:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6379], [object_number=6, tag_tid=1, entry_tid=6378]]], state=connected] 2024-09-14T06:50:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.status 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_29': 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abeb191561 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abeb191561 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:44:16 2024 2024-09-14T06:50:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:44:16 2024 2024-09-14T06:50:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:44:16 2024 2024-09-14T06:50:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abeb191561 2024-09-14T06:50:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e8f2617f-3757-4ec1-81a0-5000c4c18595 2024-09-14T06:50:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.export 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_29.export 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_29.export 2024-09-14T06:50:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.export 2024-09-14T06:50:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_29.export 2024-09-14T06:50:48.933 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:48.932+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:50:48.935 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:48.934+0000 7f14db821640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:50:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_29.export 2024-09-14T06:50:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_29.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_29.export 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-14T06:50:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-14T06:50:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-14T06:50:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:50:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:50:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:50:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-14T06:50:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-14T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_30 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_30 2024-09-14T06:50:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_30 ']' 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-09-14T06:50:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-09-14T06:50:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-14T06:50:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:50:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:50:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:50:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:50:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:50:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:50:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:50:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-09-14T06:50:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-09-14T06:50:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2024-09-14T06:50:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-09-14T06:50:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2024-09-14T06:50:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-09-14T06:50:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_30.status 2024-09-14T06:50:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.status 2024-09-14T06:50:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-09-14T06:50:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.status 2024-09-14T06:50:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:50:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6706], [object_number=7, tag_tid=1, entry_tid=6703]]], state=connected] 2024-09-14T06:50:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6706], [object_number=7, tag_tid=1, entry_tid=6703]]], state=connected] 2024-09-14T06:50:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.status 2024-09-14T06:50:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-09-14T06:50:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2024-09-14T06:50:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-09-14T06:50:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2024-09-14T06:50:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-09-14T06:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_30.status 2024-09-14T06:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.status 2024-09-14T06:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-09-14T06:50:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.status 2024-09-14T06:50:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:50:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:50:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6706], [object_number=7, tag_tid=1, entry_tid=6703]]], state=connected] 2024-09-14T06:50:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6706], [object_number=7, tag_tid=1, entry_tid=6703]]], state=connected] 2024-09-14T06:50:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.status 2024-09-14T06:50:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:50:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-14T06:50:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_30': 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0cff225f6 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0cff225f6 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:50:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:44:23 2024 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:44:23 2024 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:44:23 2024 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0cff225f6 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4153350d-f0a6-4631-81cf-b3a4c55b9562 2024-09-14T06:50:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.export 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_30.export 2024-09-14T06:50:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_30.export 2024-09-14T06:50:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.export 2024-09-14T06:50:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_30.export 2024-09-14T06:50:54.238 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:50:54.237+0000 7f14d981d640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:50:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:50:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_30.export 2024-09-14T06:50:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_30.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_30.export 2024-09-14T06:50:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:50:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-14T06:50:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-14T06:50:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:50:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-14T06:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:50:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:50:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:50:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-14T06:50:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:50:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:50:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:50:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:50:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-14T06:50:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:50:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:50:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:50:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:50:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_31 2024-09-14T06:50:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_31 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_31 ']' 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-09-14T06:50:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-09-14T06:50:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-09-14T06:50:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-09-14T06:50:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-09-14T06:50:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-14T06:50:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-14T06:50:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-14T06:50:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11894], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=11894], [object_number=13, tag_tid=1, entry_tid=11893], [object_number=12, tag_tid=1, entry_tid=11892], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-09-14T06:50:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-14T06:50:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-14T06:50:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11894], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=14, tag_tid=1, entry_tid=11894], [object_number=13, tag_tid=1, entry_tid=11893], [object_number=12, tag_tid=1, entry_tid=11892], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=14, tag_tid=1, entry_tid=11894' 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=14, tag_tid=1, entry_tid=11894' 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=11894' '!=' '' 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=14, tag_tid=1, entry_tid=11894' = '' 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=14, tag_tid=1, entry_tid=11894' 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:50:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=11894' 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:50:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=14, tag_tid=1, entry_tid=11894' 2024-09-14T06:50:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:50:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:50:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=11894 2024-09-14T06:50:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 11894 2024-09-14T06:50:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:50:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:50:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:50:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_31 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_31 ']' 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:50:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:50:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:50:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:50:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:50:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:50:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:50:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:50:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:50:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:50:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-09-14T06:50:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-09-14T06:50:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-09-14T06:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-14T06:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-14T06:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-14T06:50:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11894], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11894], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-14T06:50:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-14T06:50:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:50:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-14T06:50:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:50:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:50:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:50:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11894], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11894], [object_number=15, tag_tid=1, entry_tid=11891]]], state=connected] 2024-09-14T06:50:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.status 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-14T06:51:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_31': 2024-09-14T06:51:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab78024e0c 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab78024e0c 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:44:33 2024 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:44:33 2024 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:45:34 2024 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab78024e0c 2024-09-14T06:51:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 541f0c03-89e1-48cc-b4a9-17113ae0a79c 2024-09-14T06:51:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.export 2024-09-14T06:51:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_31.export 2024-09-14T06:51:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_31.export 2024-09-14T06:51:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.export 2024-09-14T06:51:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_31.export 2024-09-14T06:51:00.536 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:51:00.533+0000 7ff2f2001640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:51:00.536 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:51:00.533+0000 7ff2f4005640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-14T06:51:00.536 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:51:00.534+0000 7ff2f5808640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-14T06:51:00.598 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:00.596+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:51:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_31.export 2024-09-14T06:51:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_31.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_31.export 2024-09-14T06:51:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2024-09-14T06:51:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-14T06:51:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:01.203 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:51:01.201+0000 7ff305439640 -1 reset not still connected to 0x561ba4420820 2024-09-14T06:51:01.206 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:51:01.204+0000 7ff305439640 -1 reset not still connected to 0x561ba497e4e0 2024-09-14T06:51:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-14T06:51:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-14T06:51:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-14T06:51:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-14T06:51:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-14T06:51:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:51:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:51:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:51:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-14T06:51:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-14T06:51:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_32 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_32 2024-09-14T06:51:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_32 ']' 2024-09-14T06:51:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-09-14T06:51:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-14T06:51:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:51:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-09-14T06:51:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-14T06:51:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.status 2024-09-14T06:51:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-14T06:51:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.status 2024-09-14T06:51:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8619], [object_number=10, tag_tid=1, entry_tid=8618]]], state=connected] 2024-09-14T06:51:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8619], [object_number=10, tag_tid=1, entry_tid=8618]]], state=connected] 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.status 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2024-09-14T06:51:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-14T06:51:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-14T06:51:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.status 2024-09-14T06:51:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-14T06:51:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.status 2024-09-14T06:51:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8619], [object_number=10, tag_tid=1, entry_tid=8618]]], state=connected] 2024-09-14T06:51:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8619], [object_number=10, tag_tid=1, entry_tid=8618]]], state=connected] 2024-09-14T06:51:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.status 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-14T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_32': 2024-09-14T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0cf5fa6c9 2024-09-14T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0cf5fa6c9 2024-09-14T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:44:43 2024 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:44:43 2024 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:44:43 2024 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0cf5fa6c9 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: b38d35a8-d7d7-47ee-8b42-369b9f395c8c 2024-09-14T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.export 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_32.export 2024-09-14T06:51:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_32.export 2024-09-14T06:51:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.export 2024-09-14T06:51:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_32.export 2024-09-14T06:51:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_32.export 2024-09-14T06:51:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_32.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_32.export 2024-09-14T06:51:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2024-09-14T06:51:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-14T06:51:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-14T06:51:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-14T06:51:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-14T06:51:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T06:51:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T06:51:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_33 2024-09-14T06:51:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_33 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_33 ']' 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-14T06:51:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-14T06:51:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-14T06:51:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422]]], state=connected] 2024-09-14T06:51:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422], [object_number=5, tag_tid=1, entry_tid=5421], [object_number=4, tag_tid=1, entry_tid=5420]]], state=connected] 2024-09-14T06:51:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:51:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-09-14T06:51:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-14T06:51:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-14T06:51:09.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422]]], state=connected] 2024-09-14T06:51:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422], [object_number=5, tag_tid=1, entry_tid=5421], [object_number=4, tag_tid=1, entry_tid=5420]]], state=connected] 2024-09-14T06:51:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=5423' 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=5423' 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5423' '!=' '' 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5423' = '' 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=5423' 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:51:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=5423' 2024-09-14T06:51:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:51:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:51:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:51:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=5423' 2024-09-14T06:51:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=5423 2024-09-14T06:51:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 5423 2024-09-14T06:51:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_33 2024-09-14T06:51:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_33 ']' 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-09-14T06:51:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-14T06:51:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-14T06:51:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-14T06:51:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422]]], state=connected] 2024-09-14T06:51:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422]]], state=connected] 2024-09-14T06:51:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:51:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-09-14T06:51:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-14T06:51:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-14T06:51:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422]]], state=connected] 2024-09-14T06:51:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5423], [object_number=6, tag_tid=1, entry_tid=5422]]], state=connected] 2024-09-14T06:51:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.status 2024-09-14T06:51:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:51:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-14T06:51:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_33': 2024-09-14T06:51:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab8ebaf30d 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab8ebaf30d 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:44:52 2024 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:44:52 2024 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:44:52 2024 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab8ebaf30d 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 8efeeed0-7abc-4eab-adbc-f59f61281b39 2024-09-14T06:51:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.export 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_33.export 2024-09-14T06:51:11.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_33.export 2024-09-14T06:51:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.export 2024-09-14T06:51:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_33.export 2024-09-14T06:51:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_33.export 2024-09-14T06:51:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_33.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_33.export 2024-09-14T06:51:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2024-09-14T06:51:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-14T06:51:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-14T06:51:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-14T06:51:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-14T06:51:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-14T06:51:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-14T06:51:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:51:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-14T06:51:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:51:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:51:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-14T06:51:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_34 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_34 2024-09-14T06:51:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_34 ']' 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-09-14T06:51:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-14T06:51:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:51:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-09-14T06:51:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:51:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-09-14T06:51:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-09-14T06:51:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-14T06:51:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.status 2024-09-14T06:51:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-14T06:51:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.status 2024-09-14T06:51:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4959], [object_number=6, tag_tid=1, entry_tid=4958]]], state=connected] 2024-09-14T06:51:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4959], [object_number=6, tag_tid=1, entry_tid=4958]]], state=connected] 2024-09-14T06:51:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.status 2024-09-14T06:51:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-09-14T06:51:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-14T06:51:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.status 2024-09-14T06:51:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-14T06:51:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.status 2024-09-14T06:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4959], [object_number=6, tag_tid=1, entry_tid=4958]]], state=connected] 2024-09-14T06:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4959], [object_number=6, tag_tid=1, entry_tid=4958]]], state=connected] 2024-09-14T06:51:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.status 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_34': 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0e4ccc611 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0e4ccc611 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:44:59 2024 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:44:59 2024 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:46:00 2024 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0e4ccc611 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 48680c48-7556-4c60-a081-c66ad55b0266 2024-09-14T06:51:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.export 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_34.export 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_34.export 2024-09-14T06:51:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.export 2024-09-14T06:51:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_34.export 2024-09-14T06:51:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_34.export 2024-09-14T06:51:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_34.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_34.export 2024-09-14T06:51:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2024-09-14T06:51:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-14T06:51:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:16.181 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:16.179+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-14T06:51:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-14T06:51:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-14T06:51:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-14T06:51:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_35 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_35 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_35 ']' 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:19.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:19.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-09-14T06:51:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-09-14T06:51:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-09-14T06:51:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-14T06:51:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-14T06:51:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294]]], state=connected] 2024-09-14T06:51:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294], [object_number=13, tag_tid=1, entry_tid=11293], [object_number=12, tag_tid=1, entry_tid=11292]]], state=connected] 2024-09-14T06:51:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-09-14T06:51:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-09-14T06:51:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-14T06:51:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-14T06:51:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-14T06:51:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294]]], state=connected] 2024-09-14T06:51:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294], [object_number=13, tag_tid=1, entry_tid=11293], [object_number=12, tag_tid=1, entry_tid=11292]]], state=connected] 2024-09-14T06:51:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=15, tag_tid=1, entry_tid=11295' 2024-09-14T06:51:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=15, tag_tid=1, entry_tid=11295' 2024-09-14T06:51:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=15, tag_tid=1, entry_tid=11295' '!=' '' 2024-09-14T06:51:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:51:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=15, tag_tid=1, entry_tid=11295' = '' 2024-09-14T06:51:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=15, tag_tid=1, entry_tid=11295' 2024-09-14T06:51:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:51:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=15, tag_tid=1, entry_tid=11295' 2024-09-14T06:51:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:51:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:51:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=15, tag_tid=1, entry_tid=11295' 2024-09-14T06:51:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=11295 2024-09-14T06:51:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 11295 2024-09-14T06:51:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_35 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_35 ']' 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-14T06:51:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-14T06:51:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-14T06:51:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294]]], state=connected] 2024-09-14T06:51:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294]]], state=connected] 2024-09-14T06:51:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-09-14T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-09-14T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:51:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-14T06:51:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-14T06:51:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-14T06:51:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294]]], state=connected] 2024-09-14T06:51:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11295], [object_number=14, tag_tid=1, entry_tid=11294]]], state=connected] 2024-09-14T06:51:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.status 2024-09-14T06:51:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-14T06:51:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_35': 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10aba6e93b6a 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10aba6e93b6a 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:45:06 2024 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:45:06 2024 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:46:07 2024 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10aba6e93b6a 2024-09-14T06:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: ee451ea2-888d-424f-96a1-dcd46889b9eb 2024-09-14T06:51:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-14T06:51:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:51:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.export 2024-09-14T06:51:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_35.export 2024-09-14T06:51:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_35.export 2024-09-14T06:51:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.export 2024-09-14T06:51:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_35.export 2024-09-14T06:51:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_35.export 2024-09-14T06:51:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_35.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_35.export 2024-09-14T06:51:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2024-09-14T06:51:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-14T06:51:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-14T06:51:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-14T06:51:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-14T06:51:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-14T06:51:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:51:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:51:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-14T06:51:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_36 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_36 2024-09-14T06:51:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_36 ']' 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-09-14T06:51:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-14T06:51:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:51:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-09-14T06:51:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-09-14T06:51:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2024-09-14T06:51:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-09-14T06:51:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2024-09-14T06:51:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-09-14T06:51:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_36.status 2024-09-14T06:51:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.status 2024-09-14T06:51:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-09-14T06:51:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.status 2024-09-14T06:51:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4142], [object_number=7, tag_tid=1, entry_tid=4139]]], state=connected] 2024-09-14T06:51:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4142], [object_number=7, tag_tid=1, entry_tid=4139]]], state=connected] 2024-09-14T06:51:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.status 2024-09-14T06:51:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-09-14T06:51:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2024-09-14T06:51:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-09-14T06:51:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2024-09-14T06:51:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-09-14T06:51:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_36.status 2024-09-14T06:51:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.status 2024-09-14T06:51:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-09-14T06:51:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.status 2024-09-14T06:51:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4142], [object_number=7, tag_tid=1, entry_tid=4139]]], state=connected] 2024-09-14T06:51:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4142], [object_number=7, tag_tid=1, entry_tid=4139]]], state=connected] 2024-09-14T06:51:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.status 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-14T06:51:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_36': 2024-09-14T06:51:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0b10fae91 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0b10fae91 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:45:17 2024 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:45:17 2024 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:46:18 2024 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0b10fae91 2024-09-14T06:51:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 72d77723-1e5d-49d8-876d-2fdc12f5578a 2024-09-14T06:51:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-14T06:51:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:51:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.export 2024-09-14T06:51:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_36.export 2024-09-14T06:51:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_36.export 2024-09-14T06:51:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.export 2024-09-14T06:51:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_36.export 2024-09-14T06:51:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_36.export 2024-09-14T06:51:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_36.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_36.export 2024-09-14T06:51:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2024-09-14T06:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-14T06:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:26.231 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:26.230+0000 7f567c0c9640 -1 reset not still connected to 0x565521ac9110 2024-09-14T06:51:26.231 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:26.230+0000 7f567c0c9640 -1 reset not still connected to 0x565521ac96c0 2024-09-14T06:51:26.258 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:51:26.256+0000 7ff091b1f640 -1 reset not still connected to 0x55e998456b60 2024-09-14T06:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-14T06:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-14T06:51:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-14T06:51:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_37 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_37 2024-09-14T06:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_37 ']' 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-09-14T06:51:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-09-14T06:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-09-14T06:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2024-09-14T06:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-09-14T06:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2024-09-14T06:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-09-14T06:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_37.status 2024-09-14T06:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.status 2024-09-14T06:51:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-09-14T06:51:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.status 2024-09-14T06:51:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13519], [object_number=14, tag_tid=1, entry_tid=13518]]], state=connected] 2024-09-14T06:51:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13519], [object_number=14, tag_tid=1, entry_tid=13518]]], state=connected] 2024-09-14T06:51:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.status 2024-09-14T06:51:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_37.status 2024-09-14T06:51:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.status 2024-09-14T06:51:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-09-14T06:51:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.status 2024-09-14T06:51:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13519], [object_number=14, tag_tid=1, entry_tid=13518]]], state=connected] 2024-09-14T06:51:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13519], [object_number=14, tag_tid=1, entry_tid=13518]]], state=connected] 2024-09-14T06:51:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.status 2024-09-14T06:51:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-14T06:51:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_37': 2024-09-14T06:51:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab8a455704 2024-09-14T06:51:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab8a455704 2024-09-14T06:51:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:45:22 2024 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:45:22 2024 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:46:23 2024 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab8a455704 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 530c0a38-060f-47f5-9f51-d3bb03c304ea 2024-09-14T06:51:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-14T06:51:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:51:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.export 2024-09-14T06:51:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_37.export 2024-09-14T06:51:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_37.export 2024-09-14T06:51:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.export 2024-09-14T06:51:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_37.export 2024-09-14T06:51:30.107 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:30.106+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:51:30.109 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:30.108+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:51:30.172 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:30.171+0000 7f14d6016640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:51:30.172 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:30.171+0000 7f14da01e640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-14T06:51:30.177 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:30.176+0000 7f14db020640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:51:30.178 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:30.177+0000 7f14d4012640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:51:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_37.export 2024-09-14T06:51:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_37.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_37.export 2024-09-14T06:51:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2024-09-14T06:51:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-14T06:51:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:30.897 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:30.896+0000 7f566ec9d640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-09-14T06:51:30.898 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:30.897+0000 7f5669492640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2024-09-14T06:51:30.900 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:30.899+0000 7f566e49c640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 10 2024-09-14T06:51:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-14T06:51:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-14T06:51:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-14T06:51:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:51:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-14T06:51:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:51:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:51:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:51:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-14T06:51:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_38 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_38 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_38 ']' 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-09-14T06:51:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-14T06:51:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:51:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2024-09-14T06:51:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-09-14T06:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_38.status 2024-09-14T06:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.status 2024-09-14T06:51:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-09-14T06:51:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.status 2024-09-14T06:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8675], [object_number=10, tag_tid=1, entry_tid=8674]]], state=connected] 2024-09-14T06:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8675], [object_number=10, tag_tid=1, entry_tid=8674]]], state=connected] 2024-09-14T06:51:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.status 2024-09-14T06:51:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-09-14T06:51:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2024-09-14T06:51:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-09-14T06:51:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2024-09-14T06:51:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-09-14T06:51:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_38.status 2024-09-14T06:51:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.status 2024-09-14T06:51:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-09-14T06:51:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.status 2024-09-14T06:51:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8675], [object_number=10, tag_tid=1, entry_tid=8674]]], state=connected] 2024-09-14T06:51:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8675], [object_number=10, tag_tid=1, entry_tid=8674]]], state=connected] 2024-09-14T06:51:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.status 2024-09-14T06:51:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-14T06:51:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-14T06:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_38': 2024-09-14T06:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b09a87c43b 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b09a87c43b 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:45:35 2024 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:45:35 2024 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:45:35 2024 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b09a87c43b 2024-09-14T06:51:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 2185c26c-6252-4867-b055-2ac194e3f953 2024-09-14T06:51:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-14T06:51:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:51:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.export 2024-09-14T06:51:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_38.export 2024-09-14T06:51:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_38.export 2024-09-14T06:51:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.export 2024-09-14T06:51:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_38.export 2024-09-14T06:51:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_38.export 2024-09-14T06:51:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_38.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_38.export 2024-09-14T06:51:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2024-09-14T06:51:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-14T06:51:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-14T06:51:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_39 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_39 2024-09-14T06:51:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_39 ']' 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2024-09-14T06:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-14T06:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-14T06:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.status 2024-09-14T06:51:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-14T06:51:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.status 2024-09-14T06:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11863], [object_number=14, tag_tid=1, entry_tid=11862]]], state=connected] 2024-09-14T06:51:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11863], [object_number=14, tag_tid=1, entry_tid=11862]]], state=connected] 2024-09-14T06:51:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.status 2024-09-14T06:51:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-14T06:51:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.status 2024-09-14T06:51:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-14T06:51:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.status 2024-09-14T06:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11863], [object_number=14, tag_tid=1, entry_tid=11862]]], state=connected] 2024-09-14T06:51:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11863], [object_number=14, tag_tid=1, entry_tid=11862]]], state=connected] 2024-09-14T06:51:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.status 2024-09-14T06:51:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_39': 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abde34a4f2 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abde34a4f2 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:45:45 2024 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:45:45 2024 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:46:46 2024 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abde34a4f2 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 04382af0-1794-45ec-879a-36168f448319 2024-09-14T06:51:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.export 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_39.export 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_39.export 2024-09-14T06:51:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.export 2024-09-14T06:51:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_39.export 2024-09-14T06:51:40.814 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:40.813+0000 7f14d5815640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:51:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_39.export 2024-09-14T06:51:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_39.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_39.export 2024-09-14T06:51:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2024-09-14T06:51:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-14T06:51:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-14T06:51:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-14T06:51:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-14T06:51:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-14T06:51:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:51:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:51:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-14T06:51:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_40 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_40 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_40 ']' 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-09-14T06:51:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-14T06:51:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:51:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-09-14T06:51:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:51:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-09-14T06:51:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-09-14T06:51:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2024-09-14T06:51:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-09-14T06:51:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2024-09-14T06:51:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-09-14T06:51:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_40.status 2024-09-14T06:51:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.status 2024-09-14T06:51:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-09-14T06:51:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.status 2024-09-14T06:51:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4067], [object_number=6, tag_tid=1, entry_tid=4066]]], state=connected] 2024-09-14T06:51:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4067], [object_number=6, tag_tid=1, entry_tid=4066]]], state=connected] 2024-09-14T06:51:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.status 2024-09-14T06:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-09-14T06:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_40.status 2024-09-14T06:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.status 2024-09-14T06:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-09-14T06:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.status 2024-09-14T06:51:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:51:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:51:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4067], [object_number=6, tag_tid=1, entry_tid=4066]]], state=connected] 2024-09-14T06:51:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4067], [object_number=6, tag_tid=1, entry_tid=4066]]], state=connected] 2024-09-14T06:51:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.status 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-14T06:51:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_40': 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0737fb2b4 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0737fb2b4 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:45:59 2024 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:45:59 2024 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:00 2024 2024-09-14T06:51:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0737fb2b4 2024-09-14T06:51:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 2ebc569e-1eae-4032-a059-4765f7067557 2024-09-14T06:51:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.export 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_40.export 2024-09-14T06:51:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_40.export 2024-09-14T06:51:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.export 2024-09-14T06:51:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_40.export 2024-09-14T06:51:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_40.export 2024-09-14T06:51:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_40.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_40.export 2024-09-14T06:51:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2024-09-14T06:51:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-14T06:51:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-14T06:51:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-14T06:51:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-14T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_41 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_41 2024-09-14T06:51:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_41 ']' 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-09-14T06:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-09-14T06:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-09-14T06:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-14T06:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-14T06:51:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-14T06:51:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13585], [object_number=12, tag_tid=1, entry_tid=13584], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-09-14T06:51:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-14T06:51:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-14T06:51:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=13585], [object_number=12, tag_tid=1, entry_tid=13584], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=13585' 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=13585' 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=13585' '!=' '' 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=13585' = '' 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=13, tag_tid=1, entry_tid=13585' 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:51:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=13585' 2024-09-14T06:51:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:51:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:51:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=13585' 2024-09-14T06:51:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=13585 2024-09-14T06:51:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 13585 2024-09-14T06:51:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_41 2024-09-14T06:51:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_41 ']' 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-09-14T06:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-14T06:51:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-14T06:51:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-14T06:51:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-14T06:51:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-14T06:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13583], [object_number=14, tag_tid=1, entry_tid=13582]]], state=connected] 2024-09-14T06:51:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.status 2024-09-14T06:51:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-14T06:51:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_41': 2024-09-14T06:51:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab175f10cc 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab175f10cc 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:46:03 2024 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:46:03 2024 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:04 2024 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab175f10cc 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 4ea61527-1e37-4f1b-8aac-ce099146c9ff 2024-09-14T06:51:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.export 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_41.export 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_41.export 2024-09-14T06:51:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.export 2024-09-14T06:51:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_41.export 2024-09-14T06:51:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_41.export 2024-09-14T06:51:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_41.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_41.export 2024-09-14T06:51:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2024-09-14T06:51:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-14T06:51:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:50.934 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:50.933+0000 7f566d49a640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:51:50.937 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:51:50.936+0000 7f5668490640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 9 2024-09-14T06:51:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-14T06:51:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T06:51:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:51:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:51:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:51:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-14T06:51:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-14T06:51:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_42 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_42 2024-09-14T06:51:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_42 ']' 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-09-14T06:51:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T06:51:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:51:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:51:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-09-14T06:51:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:51:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:51:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:51:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:51:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:51:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:51:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-09-14T06:51:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-09-14T06:51:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_42.status 2024-09-14T06:51:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.status 2024-09-14T06:51:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-09-14T06:51:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.status 2024-09-14T06:51:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13843], [object_number=14, tag_tid=1, entry_tid=13842]]], state=connected] 2024-09-14T06:51:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13843], [object_number=14, tag_tid=1, entry_tid=13842]]], state=connected] 2024-09-14T06:51:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.status 2024-09-14T06:51:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-09-14T06:51:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2024-09-14T06:51:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-09-14T06:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2024-09-14T06:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-09-14T06:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_42.status 2024-09-14T06:51:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.status 2024-09-14T06:51:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-09-14T06:51:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.status 2024-09-14T06:51:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:51:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:51:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13843], [object_number=14, tag_tid=1, entry_tid=13842]]], state=connected] 2024-09-14T06:51:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13843], [object_number=14, tag_tid=1, entry_tid=13842]]], state=connected] 2024-09-14T06:51:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.status 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-14T06:51:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_42': 2024-09-14T06:51:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:51:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:51:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:51:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b094ae4a4f 2024-09-14T06:51:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b094ae4a4f 2024-09-14T06:51:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:51:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:51:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:51:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:51:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:46:15 2024 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:46:15 2024 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:16 2024 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b094ae4a4f 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: b53a01bd-08c9-4335-b528-a952c0361646 2024-09-14T06:51:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:51:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-14T06:51:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:51:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:51:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.export 2024-09-14T06:51:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_42.export 2024-09-14T06:51:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_42.export 2024-09-14T06:51:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.export 2024-09-14T06:51:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_42.export 2024-09-14T06:51:55.068 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:55.067+0000 7f14d4813640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:51:55.069 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:51:55.068+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:51:55.073 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-14T06:51:55.072+0000 7f641c518640 -1 bdev(0x55b7e258ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-14T06:51:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:51:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_42.export 2024-09-14T06:51:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_42.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_42.export 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:51:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-14T06:51:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-14T06:51:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:51:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:51:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-14T06:51:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:51:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:51:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:51:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:51:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:51:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:51:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:51:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:51:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:51:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:51:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-14T06:51:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:51:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:51:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:51:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:51:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:51:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:51:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_43 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:51:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:51:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_43 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_43 ']' 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-09-14T06:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-09-14T06:51:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-14T06:51:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-14T06:51:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-14T06:51:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8781], [object_number=8, tag_tid=1, entry_tid=8780], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-09-14T06:51:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-09-14T06:51:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-14T06:51:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-14T06:51:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-14T06:51:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=8781], [object_number=8, tag_tid=1, entry_tid=8780], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=8781' 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=8781' 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8781' '!=' '' 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=8781' = '' 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=8781' 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:51:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=8781' 2024-09-14T06:51:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:51:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:51:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:51:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=8781' 2024-09-14T06:51:59.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:51:59.537 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:51:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=8781 2024-09-14T06:51:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 8781 2024-09-14T06:51:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:51:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:51:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_43 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_43 ']' 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:51:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:51:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:51:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:51:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:51:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:51:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:51:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:51:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:51:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-09-14T06:51:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:51:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-14T06:51:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-14T06:51:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-14T06:51:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:51:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:51:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-14T06:51:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-14T06:51:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:51:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-14T06:51:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:51:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:51:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:51:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8779], [object_number=10, tag_tid=1, entry_tid=8778]]], state=connected] 2024-09-14T06:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.status 2024-09-14T06:51:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:51:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:51:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-14T06:51:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:51:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:51:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:51:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:51:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:51:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:51:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_43': 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab89c7f88f 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab89c7f88f 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:46:29 2024 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:46:29 2024 2024-09-14T06:52:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:30 2024 2024-09-14T06:52:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab89c7f88f 2024-09-14T06:52:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: fffbc6ca-17b7-427f-817c-f63584d269aa 2024-09-14T06:52:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.export 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_43.export 2024-09-14T06:52:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_43.export 2024-09-14T06:52:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.export 2024-09-14T06:52:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_43.export 2024-09-14T06:52:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_43.export 2024-09-14T06:52:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_43.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_43.export 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:52:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:52:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:52:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:52:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:52:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:52:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-14T06:52:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-14T06:52:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:52:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-14T06:52:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:52:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:52:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:52:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-14T06:52:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:52:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:52:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:52:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:52:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:52:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:52:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_44 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:52:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:52:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_44 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_44 ']' 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-09-14T06:52:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-09-14T06:52:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-14T06:52:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:52:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:03.506 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-09-14T06:52:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-09-14T06:52:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-09-14T06:52:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-14T06:52:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-14T06:52:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-14T06:52:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:52:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-14T06:52:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-09-14T06:52:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-14T06:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-14T06:52:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:52:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=1' '!=' '' 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=1' = '' 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:52:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=3 2024-09-14T06:52:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:52:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=2 2024-09-14T06:52:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-14T06:52:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:52:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:52:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:52:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=1 2024-09-14T06:52:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 3 = 2 -a 1 -le 1 2024-09-14T06:52:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_44 2024-09-14T06:52:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_44 ']' 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-09-14T06:52:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-14T06:52:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:52:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-09-14T06:52:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-09-14T06:52:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-14T06:52:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-14T06:52:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:52:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-14T06:52:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-09-14T06:52:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-14T06:52:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-14T06:52:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:52:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6935], [object_number=6, tag_tid=1, entry_tid=6934]]], state=connected] 2024-09-14T06:52:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.status 2024-09-14T06:52:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-14T06:52:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-14T06:52:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-14T06:52:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:52:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-14T06:52:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_44': 2024-09-14T06:52:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b0fc62fbbd 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b0fc62fbbd 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:46:39 2024 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:46:39 2024 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:40 2024 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b0fc62fbbd 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: ec957cbd-f05b-47ee-8e94-7bc5f6925ccb 2024-09-14T06:52:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.export 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_44.export 2024-09-14T06:52:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_44.export 2024-09-14T06:52:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.export 2024-09-14T06:52:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_44.export 2024-09-14T06:52:05.647 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:52:05.646+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-14T06:52:05.649 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:52:05.648+0000 7f14d901c640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-14T06:52:05.650 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:52:05.650+0000 7f14d801a640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:52:05.651 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:52:05.650+0000 7f14d5014640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-14T06:52:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_44.export 2024-09-14T06:52:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_44.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_44.export 2024-09-14T06:52:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2024-09-14T06:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-14T06:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:52:05.958 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:52:05.957+0000 7f5668c91640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 3 2024-09-14T06:52:05.962 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:52:05.961+0000 7f5668490640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 8 2024-09-14T06:52:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-14T06:52:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:52:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:52:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:52:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:52:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:52:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-14T06:52:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:52:08.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:52:08.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:52:08.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:52:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:52:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:52:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:52:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-14T06:52:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_45 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_45 2024-09-14T06:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_45 ']' 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-09-14T06:52:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_45.status 2024-09-14T06:52:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.status 2024-09-14T06:52:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-09-14T06:52:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.status 2024-09-14T06:52:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:52:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:52:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10239], [object_number=10, tag_tid=1, entry_tid=10238]]], state=connected] 2024-09-14T06:52:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10239], [object_number=10, tag_tid=1, entry_tid=10238]]], state=connected] 2024-09-14T06:52:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.status 2024-09-14T06:52:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-09-14T06:52:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_45.status 2024-09-14T06:52:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.status 2024-09-14T06:52:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-09-14T06:52:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.status 2024-09-14T06:52:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 2 2024-09-14T06:52:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:52:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10239], [object_number=10, tag_tid=1, entry_tid=10238]]], state=connected] 2024-09-14T06:52:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10239], [object_number=10, tag_tid=1, entry_tid=10238]]], state=connected] 2024-09-14T06:52:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.status 2024-09-14T06:52:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:52:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-14T06:52:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_45': 2024-09-14T06:52:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abd00b5d1c 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abd00b5d1c 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:46:47 2024 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:46:47 2024 2024-09-14T06:52:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:48 2024 2024-09-14T06:52:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abd00b5d1c 2024-09-14T06:52:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: f0206f83-0990-418f-a06a-92fa7574461b 2024-09-14T06:52:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-14T06:52:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:52:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.export 2024-09-14T06:52:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_45.export 2024-09-14T06:52:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_45.export 2024-09-14T06:52:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.export 2024-09-14T06:52:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_45.export 2024-09-14T06:52:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_45.export 2024-09-14T06:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_45.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_45.export 2024-09-14T06:52:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:52:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:11.266 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:52:11.262+0000 7ff305439640 -1 reset not still connected to 0x561bb5bf95f0 2024-09-14T06:52:11.278 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:52:11.275+0000 7ff305439640 -1 reset not still connected to 0x561bc43d12b0 2024-09-14T06:52:11.278 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:52:11.275+0000 7ff305439640 -1 reset not still connected to 0x561bd539f2b0 2024-09-14T06:52:11.278 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:52:11.275+0000 7ff305439640 -1 reset not still connected to 0x561bdb4e3860 2024-09-14T06:52:11.280 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:52:11.278+0000 7ff305439640 -1 reset not still connected to 0x561be15ea750 2024-09-14T06:52:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-14T06:52:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:52:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:52:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:52:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-14T06:52:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T06:52:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:52:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:52:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:52:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-14T06:52:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:52:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:52:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:52:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:52:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:52:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:52:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_46 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:52:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:52:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_46 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_46 ']' 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-09-14T06:52:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T06:52:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:52:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-09-14T06:52:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:52:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-09-14T06:52:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-09-14T06:52:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_46.status 2024-09-14T06:52:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.status 2024-09-14T06:52:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-09-14T06:52:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.status 2024-09-14T06:52:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5383], [object_number=6, tag_tid=1, entry_tid=5382]]], state=connected] 2024-09-14T06:52:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5383], [object_number=6, tag_tid=1, entry_tid=5382]]], state=connected] 2024-09-14T06:52:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.status 2024-09-14T06:52:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-09-14T06:52:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_46.status 2024-09-14T06:52:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.status 2024-09-14T06:52:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-09-14T06:52:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.status 2024-09-14T06:52:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5383], [object_number=6, tag_tid=1, entry_tid=5382]]], state=connected] 2024-09-14T06:52:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5383], [object_number=6, tag_tid=1, entry_tid=5382]]], state=connected] 2024-09-14T06:52:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.status 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:52:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-14T06:52:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_46': 2024-09-14T06:52:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b03a399109 2024-09-14T06:52:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b03a399109 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:46:58 2024 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:46:58 2024 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:59 2024 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b03a399109 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: fc5b649b-cd05-4ef5-9b55-ecfd5181979c 2024-09-14T06:52:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-14T06:52:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:52:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.export 2024-09-14T06:52:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_46.export 2024-09-14T06:52:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_46.export 2024-09-14T06:52:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.export 2024-09-14T06:52:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_46.export 2024-09-14T06:52:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_46.export 2024-09-14T06:52:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_46.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_46.export 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:52:16.214 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:52:16.213+0000 7f567c0c9640 -1 reset not still connected to 0x565521ac9110 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:52:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:52:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:52:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:52:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:52:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:52:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:52:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:52:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-14T06:52:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:52:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:52:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:52:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:52:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:52:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:52:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:52:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-14T06:52:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_47 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_47 2024-09-14T06:52:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_47 ']' 2024-09-14T06:52:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2024-09-14T06:52:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-09-14T06:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_47.status 2024-09-14T06:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.status 2024-09-14T06:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-09-14T06:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.status 2024-09-14T06:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11878], [object_number=15, tag_tid=1, entry_tid=11875]]], state=connected] 2024-09-14T06:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11878], [object_number=15, tag_tid=1, entry_tid=11875]]], state=connected] 2024-09-14T06:52:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.status 2024-09-14T06:52:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:52:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-09-14T06:52:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_47.status 2024-09-14T06:52:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.status 2024-09-14T06:52:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-09-14T06:52:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.status 2024-09-14T06:52:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 3 2024-09-14T06:52:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 3 2024-09-14T06:52:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11878], [object_number=15, tag_tid=1, entry_tid=11875]]], state=connected] 2024-09-14T06:52:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11878], [object_number=15, tag_tid=1, entry_tid=11875]]], state=connected] 2024-09-14T06:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.status 2024-09-14T06:52:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-14T06:52:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_47': 2024-09-14T06:52:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10abe6bdfb 2024-09-14T06:52:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10abe6bdfb 2024-09-14T06:52:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:47:04 2024 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:47:04 2024 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:48:05 2024 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10abe6bdfb 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 1db8a479-a993-4073-bc4b-a4cecfab2c09 2024-09-14T06:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-14T06:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.export 2024-09-14T06:52:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_47.export 2024-09-14T06:52:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_47.export 2024-09-14T06:52:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.export 2024-09-14T06:52:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_47.export 2024-09-14T06:52:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_47.export 2024-09-14T06:52:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_47.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_47.export 2024-09-14T06:52:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:52:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-14T06:52:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-14T06:52:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:52:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:52:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:52:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:52:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-14T06:52:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-14T06:52:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:52:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:52:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:52:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-14T06:52:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:52:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:52:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:52:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:52:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_48 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:52:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:52:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_48 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_48 ']' 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-09-14T06:52:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-09-14T06:52:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-14T06:52:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:52:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:52:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-14T06:52:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-14T06:52:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-14T06:52:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=5237], [object_number=4, tag_tid=1, entry_tid=5236], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:23.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:23.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-14T06:52:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-14T06:52:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=5237], [object_number=4, tag_tid=1, entry_tid=5236], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=5237' 2024-09-14T06:52:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=5237' 2024-09-14T06:52:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=5237' '!=' '' 2024-09-14T06:52:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T06:52:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=5237' = '' 2024-09-14T06:52:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=5237' 2024-09-14T06:52:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:52:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_tag=2 2024-09-14T06:52:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=5237' 2024-09-14T06:52:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-14T06:52:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_tag=1 2024-09-14T06:52:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:52:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_entry=1 2024-09-14T06:52:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=5237' 2024-09-14T06:52:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-14T06:52:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut -d= -f 2 2024-09-14T06:52:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_entry=5237 2024-09-14T06:52:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 2 = 1 -a 1 -le 5237 2024-09-14T06:52:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_48 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_48 ']' 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-09-14T06:52:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-14T06:52:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:52:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:52:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-09-14T06:52:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-14T06:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-14T06:52:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:52:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-14T06:52:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-14T06:52:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-14T06:52:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5235], [object_number=6, tag_tid=1, entry_tid=5234]]], state=connected] 2024-09-14T06:52:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.status 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:52:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-14T06:52:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_48': 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b09e6df402 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b09e6df402 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:47:15 2024 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:47:15 2024 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:15 2024 2024-09-14T06:52:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b09e6df402 2024-09-14T06:52:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 9e1cd5ca-0728-475c-bfe9-5ee7a145b00d 2024-09-14T06:52:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-14T06:52:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:52:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.export 2024-09-14T06:52:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_48.export 2024-09-14T06:52:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_48.export 2024-09-14T06:52:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.export 2024-09-14T06:52:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_48.export 2024-09-14T06:52:25.649 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-14T06:52:25.648+0000 7f14da81f640 -1 bdev(0x55bcae784c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-14T06:52:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_48.export 2024-09-14T06:52:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_48.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_48.export 2024-09-14T06:52:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:52:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-14T06:52:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-14T06:52:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:52:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:52:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:52:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:52:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:52:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-14T06:52:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:52:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:52:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:52:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:52:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:52:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:52:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:52:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-14T06:52:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_49 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_49 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_49 ']' 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-09-14T06:52:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-09-14T06:52:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_49.status 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.status 2024-09-14T06:52:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-09-14T06:52:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.status 2024-09-14T06:52:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-14T06:52:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-14T06:52:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.status 2024-09-14T06:52:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-09-14T06:52:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2024-09-14T06:52:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-09-14T06:52:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2024-09-14T06:52:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-09-14T06:52:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_49.status 2024-09-14T06:52:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.status 2024-09-14T06:52:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-09-14T06:52:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.status 2024-09-14T06:52:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 1 2024-09-14T06:52:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-14T06:52:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-14T06:52:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.status 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:52:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_49': 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10ab8678f1eb 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10ab8678f1eb 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:47:22 2024 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:47:22 2024 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:22 2024 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10ab8678f1eb 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: b61fc66e-fd51-4f85-a247-8c5c4150ec60 2024-09-14T06:52:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.export 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_49.export 2024-09-14T06:52:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_49.export 2024-09-14T06:52:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.export 2024-09-14T06:52:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_49.export 2024-09-14T06:52:30.022 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-14T06:52:30.021+0000 7ff2f800d640 -1 bdev(0x561ba407ec00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-14T06:52:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_49.export 2024-09-14T06:52:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_49.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_49.export 2024-09-14T06:52:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2024-09-14T06:52:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-14T06:52:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=started 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-14T06:52:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=started 2024-09-14T06:52:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local status_result 2024-09-14T06:52:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=stopped 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_instance=cluster1 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -i state 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance=cluster1 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local loop_instance 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sleep 0 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ seq 0 1 2024-09-14T06:52:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-14T06:52:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T06:52:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:0 2024-09-14T06:52:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T06:52:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T06:52:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-14T06:52:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-14T06:52:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T06:52:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ shift 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T06:52:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local local_cluster=cluster1:1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval local_cluster=cluster1 2024-09-14T06:52:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ local_cluster=cluster1 2024-09-14T06:52:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T06:52:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T06:52:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-14T06:52:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ return 0 2024-09-14T06:52:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T06:52:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo ' "state": "Replaying"' 2024-09-14T06:52:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ grep -i Replaying 2024-09-14T06:52:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: "state": "Replaying" 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=started 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test started = started 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' journal = journal ']' 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_50 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local local_cluster=cluster1 2024-09-14T06:52:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-14T06:52:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-14T06:52:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:52:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T06:52:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.2 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ flush cluster1 mirror image_50 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local 'cmd=rbd mirror flush' 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n image_50 ']' 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_instance=cluster1 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance=cluster1 2024-09-14T06:52:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loop_instance 2024-09-14T06:52:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T06:52:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-14T06:52:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:0 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T06:52:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T06:52:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T06:52:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-09-14T06:52:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-14T06:52:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd mirror flush : flush rbd mirror 2024-09-14T06:52:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:admin_socket: invalid command 2024-09-14T06:52:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1:1 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=1 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T06:52:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T06:52:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T06:52:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T06:52:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T06:52:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T06:52:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T06:52:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T06:52:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-09-14T06:52:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local id_regexp= 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_50.status 2024-09-14T06:52:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.status 2024-09-14T06:52:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-09-14T06:52:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.status 2024-09-14T06:52:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:52:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6871], [object_number=6, tag_tid=1, entry_tid=6870]]], state=connected] 2024-09-14T06:52:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6871], [object_number=6, tag_tid=1, entry_tid=6870]]], state=connected] 2024-09-14T06:52:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.status 2024-09-14T06:52:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-09-14T06:52:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2024-09-14T06:52:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-09-14T06:52:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T06:52:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:52:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2024-09-14T06:52:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local 'id_regexp=..*' 2024-09-14T06:52:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-09-14T06:52:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo cluster2-mirror-image_50.status 2024-09-14T06:52:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-14T06:52:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local status_log=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.status 2024-09-14T06:52:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-09-14T06:52:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ tee /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.status 2024-09-14T06:52:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:minimum_set: 1 2024-09-14T06:52:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:active_set: 2 2024-09-14T06:52:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:registered clients: 2024-09-14T06:52:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6871], [object_number=6, tag_tid=1, entry_tid=6870]]], state=connected] 2024-09-14T06:52:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: [id=0ec28732-cdf0-4b6c-8d3c-3b3c9c8febd1, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6871], [object_number=6, tag_tid=1, entry_tid=6870]]], state=connected] 2024-09-14T06:52:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.status 2024-09-14T06:52:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-14T06:52:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-14T06:52:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:52:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:52:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T06:52:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T06:52:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-14T06:52:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_50': 2024-09-14T06:52:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 128 MiB in 32 objects 2024-09-14T06:52:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:52:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 1 2024-09-14T06:52:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 10b023ee353d 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.10b023ee353d 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:47:29 2024 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:47:29 2024 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:47:29 2024 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: protected: False 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 10b023ee353d 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:52:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 6344cb9c-65a1-4d3c-9b58-f1d3249d16da 2024-09-14T06:52:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: false 2024-09-14T06:52:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-14T06:52:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:52:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap_name=snap 2024-09-14T06:52:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local ret=0 2024-09-14T06:52:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local rmt_export=/tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.export 2024-09-14T06:52:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local loc_export=/tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_50.export 2024-09-14T06:52:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_50.export 2024-09-14T06:52:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.export 2024-09-14T06:52:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_50.export 2024-09-14T06:52:35.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T06:52:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cmp /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_50.export 2024-09-14T06:52:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /tmp/tmp.RQ7U70XNbJ/cluster2-mirror-image_50.export /tmp/tmp.RQ7U70XNbJ/cluster1-mirror-image_50.export 2024-09-14T06:52:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: delete many images' 2024-09-14T06:52:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.RQ7U70XNbJ/rbd-mirror.test.log 2024-09-14T06:52:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2024-09-14T06:52:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2024-09-14 06:52:35 TEST: delete many images 2024-09-14T06:52:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14 06:52:35 TEST: delete many images 2024-09-14T06:52:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2024-09-14T06:52:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2024-09-14T06:52:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-14T06:52:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:52:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-14T06:52:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-14T06:52:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:52:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T06:52:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:52:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T06:52:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T06:52:36.231 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:52:36.229+0000 7ff091b1f640 -1 reset not still connected to 0x55e998457790 2024-09-14T06:52:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2024-09-14T06:52:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-14T06:52:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:52:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-14T06:52:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-14T06:52:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:52:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T06:52:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:52:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T06:52:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T06:52:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-14T06:52:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-14T06:52:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:52:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T06:52:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:52:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T06:52:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T06:52:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-14T06:52:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-14T06:52:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:52:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T06:52:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:52:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T06:52:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T06:52:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:52:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-14T06:52:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-14T06:52:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:52:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T06:52:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:52:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T06:52:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T06:52:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-14T06:52:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-14T06:52:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:52:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T06:52:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:52:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T06:52:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T06:52:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2024-09-14T06:52:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-14T06:52:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:52:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-14T06:52:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-14T06:52:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:52:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T06:52:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:52:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T06:52:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T06:52:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2024-09-14T06:52:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-14T06:52:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:52:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-14T06:52:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-14T06:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T06:52:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:52:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T06:52:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T06:52:43.370 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:52:43.369+0000 7ff091b1f640 -1 reset not still connected to 0x55e9984575f0 2024-09-14T06:52:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2024-09-14T06:52:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-14T06:52:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:52:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-14T06:52:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-14T06:52:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:52:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T06:52:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:52:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T06:52:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T06:52:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-14T06:52:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-14T06:52:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:52:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T06:52:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:52:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T06:52:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T06:52:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2024-09-14T06:52:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-14T06:52:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:52:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-14T06:52:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-14T06:52:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:52:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_11 2024-09-14T06:52:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:52:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-14T06:52:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-14T06:52:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-14T06:52:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-14T06:52:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:52:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_12 2024-09-14T06:52:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:52:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-14T06:52:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-14T06:52:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-14T06:52:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-14T06:52:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:52:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_13 2024-09-14T06:52:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:52:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-14T06:52:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-14T06:52:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-14T06:52:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-14T06:52:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:52:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_14 2024-09-14T06:52:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:52:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-14T06:52:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-14T06:52:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-14T06:52:49.447 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:52:49.446+0000 7ff0806eb640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-09-14T06:52:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-14T06:52:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:52:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_15 2024-09-14T06:52:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:52:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-14T06:52:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-14T06:52:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-14T06:52:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-14T06:52:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:52:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_16 2024-09-14T06:52:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:52:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-14T06:52:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-14T06:52:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-14T06:52:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-14T06:52:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_17 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:52:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-14T06:52:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-14T06:52:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2024-09-14T06:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-14T06:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-14T06:52:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-14T06:52:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:52:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_18 2024-09-14T06:52:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:52:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-14T06:52:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-14T06:52:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2024-09-14T06:52:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-14T06:52:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:52:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-14T06:52:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-14T06:52:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:52:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_19 2024-09-14T06:52:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:52:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-14T06:52:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-14T06:52:54.364 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:52:54.363+0000 7ff091b1f640 -1 reset not still connected to 0x55e99f223450 2024-09-14T06:52:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2024-09-14T06:52:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-14T06:52:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:52:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-14T06:52:54.878 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:52:54.876+0000 7f567c0c9640 -1 reset not still connected to 0x5655409e84e0 2024-09-14T06:52:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-14T06:52:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:52:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_20 2024-09-14T06:52:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:52:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-14T06:52:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-14T06:52:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-14T06:52:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-14T06:52:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:52:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_21 2024-09-14T06:52:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:52:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-14T06:52:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-14T06:52:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2024-09-14T06:52:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-14T06:52:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:52:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-14T06:52:56.444 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:52:56.443+0000 7ff0846f3640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2024-09-14T06:52:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-14T06:52:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:52:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_22 2024-09-14T06:52:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:52:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-14T06:52:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-14T06:52:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:52:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-14T06:52:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-14T06:52:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:52:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_23 2024-09-14T06:52:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:52:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-14T06:52:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-14T06:52:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-14T06:52:58.494 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:52:58.493+0000 7f566a494640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:52:58.494 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:52:58.493+0000 7f566d49a640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-09-14T06:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:52:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-14T06:52:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:52:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:52:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:52:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_24 2024-09-14T06:52:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:52:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-14T06:52:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-14T06:52:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:52:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:52:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:52:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2024-09-14T06:52:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-14T06:52:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:52:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:52:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:52:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:52:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-14T06:53:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-14T06:53:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:53:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_25 2024-09-14T06:53:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:53:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-14T06:53:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-14T06:53:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2024-09-14T06:53:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-14T06:53:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:53:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-14T06:53:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-14T06:53:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:53:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_26 2024-09-14T06:53:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:53:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-14T06:53:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-14T06:53:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2024-09-14T06:53:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-14T06:53:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:53:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-14T06:53:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-14T06:53:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:53:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_27 2024-09-14T06:53:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:53:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-14T06:53:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-14T06:53:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2024-09-14T06:53:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-14T06:53:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:53:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-14T06:53:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-14T06:53:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:53:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_28 2024-09-14T06:53:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:53:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-14T06:53:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-14T06:53:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2024-09-14T06:53:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-14T06:53:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:53:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-14T06:53:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-14T06:53:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:53:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_29 2024-09-14T06:53:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:53:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-14T06:53:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-14T06:53:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2024-09-14T06:53:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-14T06:53:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:53:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-14T06:53:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-14T06:53:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:53:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_30 2024-09-14T06:53:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:53:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-14T06:53:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-14T06:53:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2024-09-14T06:53:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-14T06:53:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:53:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-14T06:53:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-14T06:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:53:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_31 2024-09-14T06:53:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:53:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-14T06:53:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-14T06:53:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2024-09-14T06:53:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-14T06:53:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:53:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-14T06:53:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-14T06:53:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:53:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_32 2024-09-14T06:53:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:53:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-14T06:53:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-14T06:53:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-14T06:53:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-14T06:53:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:53:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_33 2024-09-14T06:53:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:53:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-14T06:53:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-14T06:53:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-14T06:53:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-14T06:53:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:53:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_34 2024-09-14T06:53:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:53:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-14T06:53:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-14T06:53:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2024-09-14T06:53:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-14T06:53:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:53:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-14T06:53:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-14T06:53:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:53:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_35 2024-09-14T06:53:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:53:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-14T06:53:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-14T06:53:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:53:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-14T06:53:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-14T06:53:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:53:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_36 2024-09-14T06:53:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:53:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-14T06:53:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-14T06:53:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-14T06:53:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-14T06:53:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:53:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_37 2024-09-14T06:53:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:53:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-14T06:53:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-14T06:53:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:53:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-14T06:53:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-14T06:53:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:53:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_38 2024-09-14T06:53:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:53:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-14T06:53:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-14T06:53:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2024-09-14T06:53:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-14T06:53:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:53:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-14T06:53:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-14T06:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_39 2024-09-14T06:53:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:53:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-14T06:53:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-14T06:53:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2024-09-14T06:53:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-14T06:53:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:53:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-14T06:53:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-14T06:53:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:53:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_40 2024-09-14T06:53:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:53:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-14T06:53:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-14T06:53:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2024-09-14T06:53:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-14T06:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-14T06:53:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-14T06:53:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:53:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_41 2024-09-14T06:53:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:53:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-14T06:53:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-14T06:53:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-14T06:53:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-14T06:53:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:53:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_42 2024-09-14T06:53:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:53:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-14T06:53:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-14T06:53:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-14T06:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-14T06:53:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:53:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_43 2024-09-14T06:53:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:53:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-14T06:53:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-14T06:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-14T06:53:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-14T06:53:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:53:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_44 2024-09-14T06:53:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:53:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-14T06:53:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-14T06:53:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2024-09-14T06:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-14T06:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-14T06:53:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-14T06:53:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:53:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_45 2024-09-14T06:53:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:53:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-14T06:53:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-14T06:53:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-14T06:53:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-14T06:53:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:53:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_46 2024-09-14T06:53:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:53:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-14T06:53:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-14T06:53:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2024-09-14T06:53:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-14T06:53:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:53:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-14T06:53:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-14T06:53:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:53:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_47 2024-09-14T06:53:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:53:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-14T06:53:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-14T06:53:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2024-09-14T06:53:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-14T06:53:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:53:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-14T06:53:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-14T06:53:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:53:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_48 2024-09-14T06:53:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:53:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-14T06:53:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-14T06:53:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2024-09-14T06:53:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-14T06:53:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:53:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-14T06:53:23.537 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:53:23.536+0000 7ff07feea640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-09-14T06:53:23.539 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:53:23.538+0000 7ff0846f3640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-09-14T06:53:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-14T06:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_49 2024-09-14T06:53:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:53:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-14T06:53:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-14T06:53:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2024-09-14T06:53:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-14T06:53:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:53:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local snap=snap 2024-09-14T06:53:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-14T06:53:24.751 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:53:24.749+0000 7f5669c93640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-14T06:53:24.752 INFO:tasks.ceph.cluster2.osd.0.smithi170.stderr:2024-09-14T06:53:24.751+0000 7f566d49a640 -1 bdev(0x565520734c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-14T06:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing snap: 100% complete...done. 2024-09-14T06:53:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-14T06:53:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:53:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:53:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:53:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_50 2024-09-14T06:53:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:53:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-14T06:53:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-14T06:53:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:53:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-14T06:53:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.RQ7U70XNbJ/rbd-mirror.test.log 2024-09-14T06:53:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2024-09-14T06:53:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2024-09-14 06:53:25 TEST: image deletions should propagate 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14 06:53:25 TEST: image deletions should propagate 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_count=0 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local count 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local last_count=0 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ true 2024-09-14T06:53:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2024-09-14T06:53:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2024-09-14T06:53:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2024-09-14T06:53:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T06:53:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'images: ' 2024-09-14T06:53:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 2 2024-09-14T06:53:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ count=0 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state 2024-09-14T06:53:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 40 2024-09-14T06:53:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in `seq 1 40` 2024-09-14T06:53:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 -ne 1 2024-09-14T06:53:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T06:53:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep 'image health:' 2024-09-14T06:53:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cut '-d ' -f 3 2024-09-14T06:53:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:53:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ state=OK 2024-09-14T06:53:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = ERROR 2024-09-14T06:53:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test OK = OK 2024-09-14T06:53:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 50 2024-09-14T06:53:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2024-09-14T06:53:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-14T06:53:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:53:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T06:53:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-14T06:53:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:53:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T06:53:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:53:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:53:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:53:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:53:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:53:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:53:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:53:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:53:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:53:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:53:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:53:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:53:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:53:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:53:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:53:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:53:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:53:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:53:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:53:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:53:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:53:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:53:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:53:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:53:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:53:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:53:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:53:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:53:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:53:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:53:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:53:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:53:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-14T06:53:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:53:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:53:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:53:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:53:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T06:53:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-14T06:53:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:53:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:53:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T06:53:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:53:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:53:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:53:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:53:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:53:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2024-09-14T06:53:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:53:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T06:53:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-14T06:53:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:53:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_11 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_11 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-14T06:53:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-14T06:53:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_11 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_11 2024-09-14T06:53:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_11 2024-09-14T06:53:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-14T06:53:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_12 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_12 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-14T06:53:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-14T06:53:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_12 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_12 2024-09-14T06:53:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_12 2024-09-14T06:53:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-14T06:53:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_13 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:53:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_13 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-14T06:53:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-14T06:53:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_13 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_13 2024-09-14T06:53:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_13 2024-09-14T06:53:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-14T06:53:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-14T06:53:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_14 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_14 2024-09-14T06:53:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:53:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-14T06:53:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-14T06:53:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-14T06:53:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_14 2024-09-14T06:53:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_14 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_14 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-14T06:53:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_15 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:53:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_15 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-14T06:53:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-14T06:53:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-14T06:53:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_15 2024-09-14T06:53:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_15 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_15 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-14T06:53:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-14T06:53:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_16 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_16 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-14T06:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-14T06:53:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-14T06:53:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_16 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_16 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_16 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-14T06:53:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-14T06:53:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_17 2024-09-14T06:53:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_17 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-14T06:53:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-14T06:53:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_17 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_17 2024-09-14T06:53:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_17 2024-09-14T06:53:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-14T06:53:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-14T06:53:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_18 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_18 2024-09-14T06:53:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2024-09-14T06:53:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-14T06:53:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-14T06:53:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-14T06:53:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_18 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_18 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_18 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-14T06:53:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-14T06:53:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_19 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_19 2024-09-14T06:53:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2024-09-14T06:53:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-14T06:53:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-14T06:53:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-14T06:53:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_19 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_19 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_19 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-14T06:53:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_20 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:53:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_20 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-14T06:53:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-14T06:53:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-14T06:53:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_20 2024-09-14T06:53:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_20 2024-09-14T06:53:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_20 2024-09-14T06:53:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-14T06:53:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_21 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-14T06:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:53:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_21 2024-09-14T06:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-14T06:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-14T06:53:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-14T06:53:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_21 2024-09-14T06:53:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_21 2024-09-14T06:53:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_21 2024-09-14T06:53:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-14T06:53:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_22 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_22 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2024-09-14T06:53:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-14T06:53:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-14T06:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_22 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_22 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_22 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-14T06:53:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-14T06:53:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_23 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_23 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2024-09-14T06:53:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-14T06:53:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-14T06:53:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_23 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_23 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_23 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-14T06:53:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_24 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_24 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-14T06:53:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-14T06:53:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_24 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_24 2024-09-14T06:53:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_24 2024-09-14T06:53:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-14T06:53:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-14T06:53:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_25 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_25 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-14T06:53:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-14T06:53:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_25 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_25 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_25 2024-09-14T06:53:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-14T06:53:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_26 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_26 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-14T06:53:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-14T06:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-14T06:53:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_26 2024-09-14T06:53:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_26 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_26 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-14T06:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-14T06:53:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_27 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_27 2024-09-14T06:53:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:53:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-14T06:53:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-14T06:53:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-14T06:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_27 2024-09-14T06:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_27 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_27 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-14T06:53:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-14T06:53:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_28 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_28 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-14T06:53:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-14T06:53:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_28 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_28 2024-09-14T06:53:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_28 2024-09-14T06:53:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-14T06:53:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-14T06:53:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_29 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_29 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2024-09-14T06:53:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-14T06:53:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-14T06:53:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_29 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_29 2024-09-14T06:53:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_29 2024-09-14T06:53:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-14T06:53:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_30 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:53:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_30 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-14T06:53:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-14T06:53:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_30 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_30 2024-09-14T06:53:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_30 2024-09-14T06:53:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-14T06:53:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-14T06:53:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_31 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_31 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-14T06:53:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-14T06:53:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-14T06:53:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_31 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_31 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_31 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-14T06:53:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_32 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-14T06:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_32 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-14T06:53:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-14T06:53:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_32 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_32 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_32 2024-09-14T06:53:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-14T06:53:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_33 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_33 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-14T06:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-14T06:53:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-14T06:53:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_33 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_33 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_33 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-14T06:53:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_34 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_34 2024-09-14T06:53:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2024-09-14T06:53:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-14T06:53:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-14T06:53:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-14T06:53:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_34 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_34 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_34 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-14T06:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_35 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_35 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-14T06:53:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-14T06:53:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:32.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-14T06:53:32.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.101 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_35 2024-09-14T06:53:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_35 2024-09-14T06:53:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_35 2024-09-14T06:53:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-14T06:53:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-14T06:53:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_36 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_36 2024-09-14T06:53:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2024-09-14T06:53:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-14T06:53:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-14T06:53:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-14T06:53:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_36 2024-09-14T06:53:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_36 2024-09-14T06:53:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_36 2024-09-14T06:53:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-14T06:53:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-14T06:53:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_37 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_37 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2024-09-14T06:53:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-14T06:53:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-14T06:53:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_37 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_37 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_37 2024-09-14T06:53:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-14T06:53:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_38 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_38 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-14T06:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-14T06:53:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-14T06:53:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_38 2024-09-14T06:53:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_38 2024-09-14T06:53:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_38 2024-09-14T06:53:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-14T06:53:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-14T06:53:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_39 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_39 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2024-09-14T06:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-14T06:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-14T06:53:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-14T06:53:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_39 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_39 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_39 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-14T06:53:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-14T06:53:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_40 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_40 2024-09-14T06:53:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2024-09-14T06:53:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-14T06:53:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-14T06:53:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-14T06:53:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_40 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_40 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_40 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-14T06:53:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-14T06:53:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_41 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_41 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:53:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-14T06:53:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-14T06:53:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-14T06:53:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_41 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_41 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_41 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-14T06:53:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-14T06:53:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_42 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_42 2024-09-14T06:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2024-09-14T06:53:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-14T06:53:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-14T06:53:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_42 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_42 2024-09-14T06:53:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_42 2024-09-14T06:53:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-14T06:53:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-14T06:53:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_43 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_43 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-14T06:53:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-14T06:53:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-14T06:53:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_43 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_43 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_43 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-14T06:53:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-14T06:53:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_44 2024-09-14T06:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-14T06:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_44 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-14T06:53:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-14T06:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-14T06:53:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_44 2024-09-14T06:53:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_44 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_44 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-14T06:53:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-14T06:53:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_45 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_45 2024-09-14T06:53:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2024-09-14T06:53:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-14T06:53:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-14T06:53:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-14T06:53:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_45 2024-09-14T06:53:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_45 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_45 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-14T06:53:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_46 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_46 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-14T06:53:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-14T06:53:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_46 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_46 2024-09-14T06:53:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_46 2024-09-14T06:53:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-14T06:53:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_47 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-14T06:53:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_47 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-14T06:53:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-14T06:53:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_47 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_47 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_47 2024-09-14T06:53:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-14T06:53:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-14T06:53:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_48 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_48 2024-09-14T06:53:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:53:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-14T06:53:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-14T06:53:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-14T06:53:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_48 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_48 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_48 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-14T06:53:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-14T06:53:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_49 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_49 2024-09-14T06:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2024-09-14T06:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-14T06:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-14T06:53:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-14T06:53:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_49 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_49 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_49 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-14T06:53:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-14T06:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_50 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_50 2024-09-14T06:53:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2024-09-14T06:53:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-14T06:53:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-14T06:53:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:53:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-14T06:53:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_50 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_50 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_50 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-14T06:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-14T06:53:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:53:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:53:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-14T06:53:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.RQ7U70XNbJ/rbd-mirror.test.log 2024-09-14T06:53:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2024-09-14T06:53:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2024-09-14 06:53:35 TEST: delete images during bootstrap 2024-09-14T06:53:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14 06:53:35 TEST: delete images during bootstrap 2024-09-14T06:53:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-14T06:53:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=image 2024-09-14T06:53:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-14T06:53:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:note: changing mirroring mode from pool to image 2024-09-14T06:53:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-14T06:53:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=image 2024-09-14T06:53:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-14T06:53:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:note: changing mirroring mode from pool to image 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ start_mirror cluster1 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local val=cluster1 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster_var_name=cluster 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local instance_var_name=instance 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cluster=cluster1 2024-09-14T06:53:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=cluster1 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ instance=0 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval cluster=cluster1 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ eval instance=0 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-14T06:53:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14T06:53:35.457+0000 7f5cb1874880 -1 open: failed to lock pidfile /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid because another process locked it': (11) Resource temporarily unavailable 2024-09-14T06:53:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=test 2024-09-14T06:53:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 1 10 2024-09-14T06:53:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:53:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_1 2024-09-14T06:53:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-14T06:53:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:53:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-09-14T06:53:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-14T06:53:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:53:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2024-09-14T06:53:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:53:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-14T06:53:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_1': 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1a1445b58b39 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1a1445b58b39 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:53:35 2024 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:53:35 2024 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:53:35 2024 2024-09-14T06:53:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1a1445b58b39 2024-09-14T06:53:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:53:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:53:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: ac5ffba7-611c-41d9-acd2-737dac26d9ee 2024-09-14T06:53:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:53:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-14T06:53:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:53:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=11 2024-09-14T06:53:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:53:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:53:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 28676096 bytes 2024-09-14T06:53:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 56478720 bytes 2024-09-14T06:53:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 147784704 bytes 2024-09-14T06:53:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:53:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:53:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:53:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:53:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:53:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab39cb930e 2024-09-14T06:53:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:53:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:53:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_1 present 10ab39cb930e 2024-09-14T06:53:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:53:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:53:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab39cb930e 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:53:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10ab39cb930e 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10ab39cb930e 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10ab39cb930e -o 10ab39cb930e = 10ab39cb930e 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:53:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:53:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-14T06:54:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:54:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:54:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:54:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:54:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:54:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:54:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:54:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:54:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab39cb930e 2024-09-14T06:54:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:54:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:54:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_1 deleted 10ab39cb930e 2024-09-14T06:54:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:54:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:54:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:54:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab39cb930e 2024-09-14T06:54:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:54:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:54:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:54:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:54:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:54:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_1 2024-09-14T06:54:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:54:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:54:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T06:54:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:54:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:54:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:54:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:54:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-14T06:54:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:54:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:54:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-14T06:54:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-14T06:54:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:54:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:54:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:54:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:54:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T06:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_1 2024-09-14T06:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T06:54:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T06:54:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:54:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_2 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:54:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:54:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-09-14T06:54:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-14T06:54:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:54:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:54:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:54:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2024-09-14T06:54:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:54:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_2': 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1a2c2a16ac7e 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1a2c2a16ac7e 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:54:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:54:18 2024 2024-09-14T06:54:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:54:18 2024 2024-09-14T06:54:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:54:18 2024 2024-09-14T06:54:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1a2c2a16ac7e 2024-09-14T06:54:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:54:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:54:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: d52e644a-8184-49f4-b65e-5fb497ecfe2d 2024-09-14T06:54:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:54:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-14T06:54:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:54:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:54:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:54:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:54:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:54:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:54:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 33813504 bytes 2024-09-14T06:54:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 127134720 bytes 2024-09-14T06:54:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 140226560 bytes 2024-09-14T06:54:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 203304960 bytes 2024-09-14T06:54:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 207680512 bytes 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:54:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10b0bcbb0fba 2024-09-14T06:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:54:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:54:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_2 present 10b0bcbb0fba 2024-09-14T06:54:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:54:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:54:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:54:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10b0bcbb0fba 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:54:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:54:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10b0bcbb0fba 2024-09-14T06:54:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10b0bcbb0fba 2024-09-14T06:54:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10b0bcbb0fba -o 10b0bcbb0fba = 10b0bcbb0fba 2024-09-14T06:54:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:54:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:54:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:54:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-14T06:54:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:54:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:54:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:54:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-14T06:55:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:55:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-14T06:55:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:55:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:55:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:55:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:55:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:55:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:55:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:55:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:55:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:55:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:55:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:55:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:55:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:55:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:55:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:55:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_2 2024-09-14T06:55:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:55:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:55:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:55:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-14T06:55:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-14T06:55:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:55:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:55:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:55:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T06:55:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_2 2024-09-14T06:55:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T06:55:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T06:55:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_3 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:55:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:55:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-09-14T06:55:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-14T06:55:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:55:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2024-09-14T06:55:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:55:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_3': 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1a44122d397f 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1a44122d397f 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:55:02 2024 2024-09-14T06:55:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:55:02 2024 2024-09-14T06:55:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:55:02 2024 2024-09-14T06:55:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1a44122d397f 2024-09-14T06:55:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:55:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:55:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 0284fa47-2feb-4dce-8612-f4e4216104fc 2024-09-14T06:55:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:55:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-14T06:55:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:55:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:55:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:55:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:55:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 6480896 bytes 2024-09-14T06:55:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 36934656 bytes 2024-09-14T06:55:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 49783808 bytes 2024-09-14T06:55:07.485 INFO:tasks.ceph.cluster2.osd.1.smithi170.stderr:2024-09-14T06:55:07.484+0000 7f0c69033640 -1 reset not still connected to 0x562d149ee000 2024-09-14T06:55:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 94134272 bytes 2024-09-14T06:55:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 104144896 bytes 2024-09-14T06:55:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 126402560 bytes 2024-09-14T06:55:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:700 IOs, 202635264 bytes 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:55:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:55:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:55:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab3bc0ffcc 2024-09-14T06:55:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:55:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:55:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 present 10ab3bc0ffcc 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab3bc0ffcc 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:55:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:55:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10ab3bc0ffcc 2024-09-14T06:55:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10ab3bc0ffcc 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10ab3bc0ffcc -o 10ab3bc0ffcc = 10ab3bc0ffcc 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-14T06:55:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:55:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:55:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:55:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:55:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:55:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:55:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:55:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:55:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:55:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:55:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:55:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_3 2024-09-14T06:55:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:55:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:55:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-14T06:55:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-14T06:55:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:55:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:55:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T06:55:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_3 2024-09-14T06:55:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T06:55:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T06:55:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_4 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:55:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:55:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:55:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:55:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-09-14T06:55:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-14T06:55:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:55:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:55:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2024-09-14T06:55:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:55:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-14T06:55:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_4': 2024-09-14T06:55:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:55:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:55:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1a5cfe229bc8 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1a5cfe229bc8 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:55:46 2024 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:55:46 2024 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:55:46 2024 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1a5cfe229bc8 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e7450b9d-456e-410b-8fdb-7d21dccd324e 2024-09-14T06:55:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:55:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-14T06:55:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:55:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:55:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:55:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:55:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:55:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:55:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 27536384 bytes 2024-09-14T06:55:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 80184320 bytes 2024-09-14T06:55:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 120997888 bytes 2024-09-14T06:55:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 187472896 bytes 2024-09-14T06:55:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:55:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:55:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:55:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:55:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:55:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10abf94fedf6 2024-09-14T06:55:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:55:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:56:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_4 present 10abf94fedf6 2024-09-14T06:56:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:56:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:56:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:56:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10abf94fedf6 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:56:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:56:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10abf94fedf6 2024-09-14T06:56:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10abf94fedf6 2024-09-14T06:56:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10abf94fedf6 -o 10abf94fedf6 = 10abf94fedf6 2024-09-14T06:56:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:56:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:56:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:56:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-14T06:56:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:56:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-14T06:56:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:56:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-14T06:56:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:56:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:56:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:56:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:56:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:56:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:56:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:56:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_4 2024-09-14T06:56:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:56:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:56:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:56:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:56:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:56:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:56:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:56:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-14T06:56:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:56:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-14T06:56:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-14T06:56:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:56:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:56:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T06:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_4 2024-09-14T06:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T06:56:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T06:56:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:56:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:56:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:56:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_5 2024-09-14T06:56:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-14T06:56:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:56:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-09-14T06:56:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-14T06:56:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:56:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:56:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2024-09-14T06:56:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:56:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-14T06:56:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_5': 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1a745a58584e 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1a745a58584e 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:56:30 2024 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:56:30 2024 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:56:30 2024 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1a745a58584e 2024-09-14T06:56:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:56:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:56:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 57210f8a-cfa5-4d08-a891-d6056e8d7773 2024-09-14T06:56:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:56:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-14T06:56:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:56:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:56:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2024-09-14T06:56:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:56:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:56:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 3329024 bytes 2024-09-14T06:56:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 28396544 bytes 2024-09-14T06:56:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 73491456 bytes 2024-09-14T06:56:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 80028672 bytes 2024-09-14T06:56:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 106299392 bytes 2024-09-14T06:56:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 139458560 bytes 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:56:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:56:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:56:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab63326e4c 2024-09-14T06:56:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:56:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:56:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_5 present 10ab63326e4c 2024-09-14T06:56:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:56:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:56:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab63326e4c 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:56:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10ab63326e4c 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10ab63326e4c 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10ab63326e4c -o 10ab63326e4c = 10ab63326e4c 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:56:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:56:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:56:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-14T06:57:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:57:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-14T06:57:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:57:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:57:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:57:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:57:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:57:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:57:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:57:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_5 2024-09-14T06:57:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:57:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:57:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:57:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:57:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-14T06:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:57:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-14T06:57:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-14T06:57:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:57:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:57:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:57:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T06:57:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_5 2024-09-14T06:57:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T06:57:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T06:57:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:57:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:57:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:57:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_6 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-09-14T06:57:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-14T06:57:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:57:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2024-09-14T06:57:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:57:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_6': 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1a8c83e2df32 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1a8c83e2df32 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:57:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:57:12 2024 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:57:12 2024 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:57:12 2024 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1a8c83e2df32 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 14832695-54bc-48c3-ae31-37f1a9f3ed09 2024-09-14T06:57:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:57:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-14T06:57:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:57:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=13 2024-09-14T06:57:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:57:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:57:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 37224448 bytes 2024-09-14T06:57:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 92476416 bytes 2024-09-14T06:57:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 126793728 bytes 2024-09-14T06:57:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 181749760 bytes 2024-09-14T06:57:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 211451904 bytes 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:57:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:57:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab80bd6cc5 2024-09-14T06:57:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:57:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:57:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_6 present 10ab80bd6cc5 2024-09-14T06:57:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:57:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:57:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab80bd6cc5 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:57:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10ab80bd6cc5 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10ab80bd6cc5 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10ab80bd6cc5 -o 10ab80bd6cc5 = 10ab80bd6cc5 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-14T06:57:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:57:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:57:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:57:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:57:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:57:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:57:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:57:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:57:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:57:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_6 2024-09-14T06:57:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:57:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:57:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:57:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:57:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-14T06:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-14T06:57:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-14T06:57:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:57:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:57:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T06:57:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_6 2024-09-14T06:57:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T06:57:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T06:57:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:57:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_7 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:57:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:57:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:57:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:57:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-09-14T06:57:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-14T06:57:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:57:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:57:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2024-09-14T06:57:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:57:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-14T06:57:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_7': 2024-09-14T06:57:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:57:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:57:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:57:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1aa4e8cadebe 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1aa4e8cadebe 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:57:55 2024 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:57:55 2024 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:57:55 2024 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1aa4e8cadebe 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: aaec30be-3f07-4c92-907e-c59b9460a133 2024-09-14T06:57:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:57:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-14T06:57:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:57:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:57:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:57:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:57:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=10 2024-09-14T06:57:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:57:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:57:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 631808 bytes 2024-09-14T06:57:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 13552640 bytes 2024-09-14T06:57:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 28667904 bytes 2024-09-14T06:58:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 80033792 bytes 2024-09-14T06:58:02.802 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:58:02.801+0000 7ff07eee8640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-09-14T06:58:02.808 INFO:tasks.ceph.cluster2.osd.2.smithi170.stderr:2024-09-14T06:58:02.807+0000 7ff083ef2640 -1 bdev(0x55e9972d2c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2024-09-14T06:58:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 111472640 bytes 2024-09-14T06:58:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:600 IOs, 112087040 bytes 2024-09-14T06:58:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:58:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:58:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:58:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:58:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:58:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:58:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:58:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:58:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:58:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10abbf8e7811 2024-09-14T06:58:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:58:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 present 10abbf8e7811 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10abbf8e7811 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:58:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:58:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:58:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:58:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:58:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:58:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10abbf8e7811 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10abbf8e7811 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10abbf8e7811 -o 10abbf8e7811 = 10abbf8e7811 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-14T06:58:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:58:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-14T06:58:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:58:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:58:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:58:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:58:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:58:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:58:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:58:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_7 2024-09-14T06:58:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:58:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:58:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-14T06:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-14T06:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:58:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T06:58:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_7 2024-09-14T06:58:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T06:58:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T06:58:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_8 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:58:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:58:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:58:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-09-14T06:58:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-14T06:58:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:58:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:58:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2024-09-14T06:58:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:58:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-14T06:58:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_8': 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1abbbb8d5610 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1abbbb8d5610 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:58:36 2024 2024-09-14T06:58:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:58:36 2024 2024-09-14T06:58:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:58:36 2024 2024-09-14T06:58:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1abbbb8d5610 2024-09-14T06:58:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:58:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:58:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: e2d530c1-1ba0-4be7-b23c-6ccf553ffd62 2024-09-14T06:58:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:58:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-14T06:58:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:58:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:58:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2024-09-14T06:58:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:58:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:58:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 43256832 bytes 2024-09-14T06:58:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:58:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:58:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:58:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:58:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:58:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:58:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:58:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:58:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab297f08de 2024-09-14T06:58:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:58:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:58:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_8 present 10ab297f08de 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab297f08de 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:58:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:58:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10ab297f08de 2024-09-14T06:58:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10ab297f08de 2024-09-14T06:58:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10ab297f08de -o 10ab297f08de = 10ab297f08de 2024-09-14T06:58:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:58:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:58:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:58:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-14T06:58:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:58:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:58:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:58:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-14T06:59:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:59:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-14T06:59:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:59:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:59:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:59:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:59:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab297f08de 2024-09-14T06:59:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:59:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_8 deleted 10ab297f08de 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab297f08de 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:59:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:59:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:59:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:59:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_8 2024-09-14T06:59:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:59:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:59:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T06:59:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:59:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:59:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:59:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:59:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-14T06:59:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:59:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-14T06:59:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-14T06:59:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:59:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:59:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:59:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T06:59:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_8 2024-09-14T06:59:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T06:59:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T06:59:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_9 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:59:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:59:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:59:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-09-14T06:59:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-14T06:59:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:59:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2024-09-14T06:59:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:59:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-14T06:59:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_9': 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1ad31f873db8 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1ad31f873db8 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:59:14 2024 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:59:14 2024 2024-09-14T06:59:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:59:14 2024 2024-09-14T06:59:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1ad31f873db8 2024-09-14T06:59:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:59:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:59:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: 2a2e1246-4145-4f82-a956-4121f3a56ef2 2024-09-14T06:59:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:59:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-14T06:59:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:59:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=12 2024-09-14T06:59:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:59:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:59:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 51430400 bytes 2024-09-14T06:59:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:200 IOs, 76187648 bytes 2024-09-14T06:59:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:300 IOs, 100714496 bytes 2024-09-14T06:59:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:400 IOs, 157784064 bytes 2024-09-14T06:59:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:500 IOs, 189022208 bytes 2024-09-14T06:59:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:59:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:59:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:59:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:59:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:59:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:59:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:59:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10b0210da3f3 2024-09-14T06:59:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:59:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_9 present 10b0210da3f3 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10b0210da3f3 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:59:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:59:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:59:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10b0210da3f3 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10b0210da3f3 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10b0210da3f3 -o 10b0210da3f3 = 10b0210da3f3 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-14T06:59:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T06:59:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-14T06:59:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:59:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:59:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:59:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id= 2024-09-14T06:59:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:59:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T06:59:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T06:59:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_9 2024-09-14T06:59:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:59:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:59:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-14T06:59:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-14T06:59:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:59:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T06:59:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T06:59:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_9 2024-09-14T06:59:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T06:59:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T06:59:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for i in `seq 1 10` 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image=image_10 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local size=128 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 512M ']' 2024-09-14T06:59:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ size=512M 2024-09-14T06:59:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ shift 2024-09-14T06:59:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-09-14T06:59:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-14T06:59:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:59:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local mode=journal 2024-09-14T06:59:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2024-09-14T06:59:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring enabled 2024-09-14T06:59:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-14T06:59:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:rbd image 'image_10': 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: size 512 MiB in 128 objects 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: order 22 (4 MiB objects) 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: snapshot_count: 0 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: id: 1aeb33ba8ab9 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: block_name_prefix: rbd_data.1aeb33ba8ab9 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: format: 2 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: features: layering, exclusive-lock, journaling 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: op_features: 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: flags: 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: create_timestamp: Sat Sep 14 06:59:55 2024 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: access_timestamp: Sat Sep 14 06:59:55 2024 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: modify_timestamp: Sat Sep 14 06:59:55 2024 2024-09-14T06:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: journal: 1aeb33ba8ab9 2024-09-14T06:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring state: enabled 2024-09-14T06:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring mode: journal 2024-09-14T06:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring global id: da0d4f84-5674-46d5-a16d-380ec3400abd 2024-09-14T06:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout: mirroring primary: true 2024-09-14T06:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-14T06:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T06:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T06:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T06:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:59:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local duration=5 2024-09-14T06:59:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T06:59:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:59:58.044 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:100 IOs, 38461440 bytes 2024-09-14T07:00:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=124 2024-09-14T07:00:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T07:00:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-14T07:00:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=present 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T07:00:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T07:00:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab867579e3 2024-09-14T07:00:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T07:00:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T07:00:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_10 present 10ab867579e3 2024-09-14T07:00:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=present 2024-09-14T07:00:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab867579e3 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T07:00:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id=10ab867579e3 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n 10ab867579e3 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -z 10ab867579e3 -o 10ab867579e3 = 10ab867579e3 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_state=present 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test present = present 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-14T07:00:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-14T07:00:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:Mirroring disabled 2024-09-14T07:00:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local state=deleted 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id= 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local s 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T07:00:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T07:00:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ image_id=10ab867579e3 2024-09-14T07:00:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T07:00:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0.1 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test_image_present cluster1 mirror image_10 deleted 10ab867579e3 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local test_state=deleted 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image_id=10ab867579e3 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_state=deleted 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local current_image_id 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local image=image_10 2024-09-14T07:00:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T07:00:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T07:00:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ current_image_id= 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test deleted = deleted 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-14T07:00:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-14T07:00:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T07:00:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T07:00:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T07:00:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local image=image_10 2024-09-14T07:00:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T07:00:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T07:00:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T07:00:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-14T07:00:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ tee -a /tmp/tmp.RQ7U70XNbJ/rbd-mirror.test.log 2024-09-14T07:00:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ date '+%F %T' 2024-09-14T07:00:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo 2024-09-14 07:00:36 TEST: check if removed 'images'\''' OMAP are removed 2024-09-14T07:00:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:2024-09-14 07:00:36 TEST: check if removed images' OMAP are removed 2024-09-14T07:00:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-14T07:00:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=status_global 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T07:00:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-14T07:00:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=status_global 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-14T07:00:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c status_global 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2024-09-14T07:00:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_ 2024-09-14T07:00:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T07:00:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-14T07:00:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_ 2024-09-14T07:00:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-14T07:00:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_ 2024-09-14T07:00:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirror_leader 2024-09-14T07:00:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_map 2024-09-14T07:00:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T07:00:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-14T07:00:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2024-09-14T07:00:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-14T07:00:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T07:00:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2024-09-14T07:00:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 = 0 2024-09-14T07:00:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=1 2024-09-14T07:00:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 -eq 0 ']' 2024-09-14T07:00:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2024-09-14T07:00:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T07:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 1 = 0 2024-09-14T07:00:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=1 2024-09-14T07:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 1 -eq 0 ']' 2024-09-14T07:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2024-09-14T07:00:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-14T07:00:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2024-09-14T07:00:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-14T07:00:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2024-09-14T07:00:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T07:00:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2024-09-14T07:00:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T07:00:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=status_global 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T07:00:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=status_global 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-14T07:00:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c status_global 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirroring 2024-09-14T07:00:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_ 2024-09-14T07:00:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T07:00:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-14T07:00:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_ 2024-09-14T07:00:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-14T07:00:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirroring 2024-09-14T07:00:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_ 2024-09-14T07:00:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pool=mirror 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local obj_name=rbd_mirror_leader 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local filter=image_map 2024-09-14T07:00:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 0 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local filter=image_map 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local pool=mirror 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T07:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ grep -c image_map 2024-09-14T07:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test 0 = 0 2024-09-14T07:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ error_code=0 2024-09-14T07:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set -e 2024-09-14T07:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T07:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ return 0 2024-09-14T07:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ cleanup 0 2024-09-14T07:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local error_code=0 2024-09-14T07:00:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ set +e 2024-09-14T07:00:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -ne 0 ']' 2024-09-14T07:00:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z '' ']' 2024-09-14T07:00:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster instance 2024-09-14T07:00:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T07:00:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-14T07:00:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' does not exist 2024-09-14T07:00:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T07:00:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-14T07:00:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror' does not exist 2024-09-14T07:00:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T07:00:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-14T07:00:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' does not exist 2024-09-14T07:00:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ CEPH_ARGS= 2024-09-14T07:00:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-14T07:00:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:pool 'mirror_parent' does not exist 2024-09-14T07:00:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-14T07:00:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirrors cluster1 2024-09-14T07:00:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1 2024-09-14T07:00:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2024-09-14T07:00:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T07:00:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T07:00:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster1:0 '' 2024-09-14T07:00:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:0 2024-09-14T07:00:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2024-09-14T07:00:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T07:00:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pid 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_pid_file cluster1:0 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1:0 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:0 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=cluster 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T07:00:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval cluster=cluster1 2024-09-14T07:00:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ cluster=cluster1 2024-09-14T07:00:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T07:00:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T07:00:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-14T07:00:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-14T07:00:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-14T07:00:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ pid=38472 2024-09-14T07:00:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 38472 ']' 2024-09-14T07:00:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ kill 38472 2024-09-14T07:00:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T07:00:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T07:00:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38472 '$2 == pid {print; exit 1}' 2024-09-14T07:00:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:ubuntu 38472 10.7 0.2 2014000 88576 ? Ssl 06:30 3:15 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-14T07:00:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T07:00:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 2 2024-09-14T07:00:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38472 '$2 == pid {print; exit 1}' 2024-09-14T07:00:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T07:00:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38472 '$2 == pid {print; exit 1}' 2024-09-14T07:00:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-14T07:00:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T07:00:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T07:00:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T07:00:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:0 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T07:00:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T07:00:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T07:00:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T07:00:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T07:00:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T07:00:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T07:00:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_pid_file cluster1:0 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:0 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:0 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T07:00:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T07:00:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T07:00:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T07:00:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T07:00:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-14T07:00:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-14T07:00:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-14T07:00:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T07:00:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster1:1 '' 2024-09-14T07:00:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster1:1 2024-09-14T07:00:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2024-09-14T07:00:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T07:00:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pid 2024-09-14T07:00:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_pid_file cluster1:1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster1:1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster1:1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=cluster 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval cluster=cluster1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ cluster=cluster1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T07:00:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T07:00:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-14T07:00:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-14T07:00:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-14T07:00:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ pid=38516 2024-09-14T07:00:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 38516 ']' 2024-09-14T07:00:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ kill 38516 2024-09-14T07:00:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T07:00:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T07:00:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38516 '$2 == pid {print; exit 1}' 2024-09-14T07:00:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T07:00:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38516 '$2 == pid {print; exit 1}' 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T07:00:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T07:00:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T07:00:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-09-14T07:00:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster1:1 2024-09-14T07:00:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T07:00:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T07:00:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster1 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster1 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T07:00:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T07:00:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_pid_file cluster1:1 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1:1 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster1:1 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster1 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T07:00:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T07:00:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster1 2024-09-14T07:00:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster1 2024-09-14T07:00:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T07:00:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-14T07:00:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-14T07:00:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-14T07:00:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-14T07:00:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirrors cluster2 2024-09-14T07:00:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2 2024-09-14T07:00:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2024-09-14T07:00:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ seq 0 1 2024-09-14T07:00:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T07:00:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster2:0 '' 2024-09-14T07:00:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:0 2024-09-14T07:00:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2024-09-14T07:00:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T07:00:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pid 2024-09-14T07:00:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_pid_file cluster2:0 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster2:0 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster2:0 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=cluster 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster2 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval cluster=cluster2 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ cluster=cluster2 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=0 2024-09-14T07:00:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=0 2024-09-14T07:00:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-14T07:00:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-14T07:00:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-14T07:00:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ pid=38577 2024-09-14T07:00:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 38577 ']' 2024-09-14T07:00:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ kill 38577 2024-09-14T07:00:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T07:00:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T07:00:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38577 '$2 == pid {print; exit 1}' 2024-09-14T07:00:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T07:00:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38577 '$2 == pid {print; exit 1}' 2024-09-14T07:00:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster2:0 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster2:0 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster2 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster2 2024-09-14T07:00:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T07:00:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-14T07:00:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-14T07:00:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-14T07:00:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-14T07:00:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster2:0 2024-09-14T07:00:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-14T07:00:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster2:0 2024-09-14T07:00:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster2 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster2 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-14T07:00:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-14T07:00:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_pid_file cluster2:0 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2:0 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster2:0 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=0 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-14T07:00:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster2 2024-09-14T07:00:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster2 2024-09-14T07:00:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=0 2024-09-14T07:00:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=0 2024-09-14T07:00:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-14T07:00:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-14T07:00:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-14T07:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T07:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ stop_mirror cluster2:1 '' 2024-09-14T07:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local cluster=cluster2:1 2024-09-14T07:00:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local sig= 2024-09-14T07:00:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test -n '' 2024-09-14T07:00:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ local pid 2024-09-14T07:00:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ daemon_pid_file cluster2:1 2024-09-14T07:00:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster=cluster2:1 2024-09-14T07:00:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local val=cluster2:1 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local cluster_var_name=cluster 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local instance_var_name=instance 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster2 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval cluster=cluster2 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ cluster=cluster2 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ eval instance=1 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ instance=1 2024-09-14T07:00:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-14T07:00:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-14T07:00:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-14T07:00:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ pid=38639 2024-09-14T07:00:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -n 38639 ']' 2024-09-14T07:00:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ kill 38639 2024-09-14T07:00:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T07:00:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ sleep 1 2024-09-14T07:00:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38639 '$2 == pid {print; exit 1}' 2024-09-14T07:00:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ break 2024-09-14T07:00:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ ps auxww 2024-09-14T07:00:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ awk -v pid=38639 '$2 == pid {print; exit 1}' 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster2:1 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster1 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster2:1 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-14T07:00:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster2 2024-09-14T07:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster2 2024-09-14T07:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T07:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-14T07:00:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-14T07:00:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local local_cluster=cluster2:1 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster2:1 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=local_cluster 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T07:00:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-14T07:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval local_cluster=cluster2 2024-09-14T07:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ local_cluster=cluster2 2024-09-14T07:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T07:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-14T07:00:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-14T07:00:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-14T07:00:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ daemon_pid_file cluster2:1 2024-09-14T07:00:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster=cluster2:1 2024-09-14T07:00:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local val=cluster2:1 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local cluster_var_name=cluster 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ local instance_var_name=instance 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ cluster=cluster2 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ instance=1 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval cluster=cluster2 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ cluster=cluster2 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ eval instance=1 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ instance=1 2024-09-14T07:00:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-14T07:00:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-14T07:00:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-14T07:00:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' -z 1 ']' 2024-09-14T07:00:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ test '' = /tmp/tmp.RQ7U70XNbJ 2024-09-14T07:00:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ rm -Rf /tmp/tmp.RQ7U70XNbJ 2024-09-14T07:00:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T07:00:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ echo OK 2024-09-14T07:00:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stdout:OK 2024-09-14T07:00:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi170.stderr:+ exit 0 2024-09-14T07:00:50.027 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-14T07:00:50.028 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-14T07:00:50.227 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-14T07:00:50.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-14T07:00:50.513 DEBUG:teuthology.parallel:result is None 2024-09-14T07:00:50.513 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T07:00:50.538 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T07:00:50.539 DEBUG:teuthology.orchestra.run.smithi170:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T07:00:50.595 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T07:00:50.595 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-14T07:00:50.611 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-14T07:00:50.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T07:00:50.923 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T07:00:50.924 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-09-14T07:00:50.936 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1370,"stamp":"2024-09-14T07:00:49.956837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3013512,"kb_used_data":15436,"kb_used_omap":377,"kb_used_meta":2997638,"kb_avail":280102008,"statfs":{"total":289910292480,"available":286824456192,"internally_reserved":0,"allocated":15806464,"data_stored":8026078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":386176,"internal_metadata":3069582208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-1963,"num_read_kb":-1952,"num_write":-132,"num_write_kb":-129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.773526"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194187+0000","last_change":"2024-09-14T06:28:54.782642+0000","last_active":"2024-09-14T07:00:43.194187+0000","last_peered":"2024-09-14T07:00:43.194187+0000","last_clean":"2024-09-14T07:00:43.194187+0000","last_became_active":"2024-09-14T06:28:52.776292+0000","last_became_peered":"2024-09-14T06:28:52.776292+0000","last_unstale":"2024-09-14T07:00:43.194187+0000","last_undegraded":"2024-09-14T07:00:43.194187+0000","last_fullsized":"2024-09-14T07:00:43.194187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:48:57.788740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010453089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192722+0000","last_change":"2024-09-14T06:28:54.782290+0000","last_active":"2024-09-14T07:00:43.192722+0000","last_peered":"2024-09-14T07:00:43.192722+0000","last_clean":"2024-09-14T07:00:43.192722+0000","last_became_active":"2024-09-14T06:28:52.776371+0000","last_became_peered":"2024-09-14T06:28:52.776371+0000","last_unstale":"2024-09-14T07:00:43.192722+0000","last_undegraded":"2024-09-14T07:00:43.192722+0000","last_fullsized":"2024-09-14T07:00:43.192722+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:49:06.151379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077233399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194838+0000","last_change":"2024-09-14T06:28:54.782567+0000","last_active":"2024-09-14T07:00:43.194838+0000","last_peered":"2024-09-14T07:00:43.194838+0000","last_clean":"2024-09-14T07:00:43.194838+0000","last_became_active":"2024-09-14T06:28:52.776250+0000","last_became_peered":"2024-09-14T06:28:52.776250+0000","last_unstale":"2024-09-14T07:00:43.194838+0000","last_undegraded":"2024-09-14T07:00:43.194838+0000","last_fullsized":"2024-09-14T07:00:43.194838+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:21:07.454036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011012839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193506+0000","last_change":"2024-09-14T06:28:54.781991+0000","last_active":"2024-09-14T07:00:43.193506+0000","last_peered":"2024-09-14T07:00:43.193506+0000","last_clean":"2024-09-14T07:00:43.193506+0000","last_became_active":"2024-09-14T06:28:52.776740+0000","last_became_peered":"2024-09-14T06:28:52.776740+0000","last_unstale":"2024-09-14T07:00:43.193506+0000","last_undegraded":"2024-09-14T07:00:43.193506+0000","last_fullsized":"2024-09-14T07:00:43.193506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:17:00.207670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064401499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.309808+0000","last_change":"2024-09-14T06:28:54.780145+0000","last_active":"2024-09-14T07:00:43.309808+0000","last_peered":"2024-09-14T07:00:43.309808+0000","last_clean":"2024-09-14T07:00:43.309808+0000","last_became_active":"2024-09-14T06:28:52.776357+0000","last_became_peered":"2024-09-14T06:28:52.776357+0000","last_unstale":"2024-09-14T07:00:43.309808+0000","last_undegraded":"2024-09-14T07:00:43.309808+0000","last_fullsized":"2024-09-14T07:00:43.309808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:57:29.100954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096998400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.402626+0000","last_change":"2024-09-14T06:28:54.777770+0000","last_active":"2024-09-14T07:00:43.402626+0000","last_peered":"2024-09-14T07:00:43.402626+0000","last_clean":"2024-09-14T07:00:43.402626+0000","last_became_active":"2024-09-14T06:28:52.777933+0000","last_became_peered":"2024-09-14T06:28:52.777933+0000","last_unstale":"2024-09-14T07:00:43.402626+0000","last_undegraded":"2024-09-14T07:00:43.402626+0000","last_fullsized":"2024-09-14T07:00:43.402626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:50.657246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049603799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.404434+0000","last_change":"2024-09-14T06:28:54.777772+0000","last_active":"2024-09-14T07:00:43.404434+0000","last_peered":"2024-09-14T07:00:43.404434+0000","last_clean":"2024-09-14T07:00:43.404434+0000","last_became_active":"2024-09-14T06:28:52.777950+0000","last_became_peered":"2024-09-14T06:28:52.777950+0000","last_unstale":"2024-09-14T07:00:43.404434+0000","last_undegraded":"2024-09-14T07:00:43.404434+0000","last_fullsized":"2024-09-14T07:00:43.404434+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:44:17.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504179,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193294+0000","last_change":"2024-09-14T06:28:54.782038+0000","last_active":"2024-09-14T07:00:43.193294+0000","last_peered":"2024-09-14T07:00:43.193294+0000","last_clean":"2024-09-14T07:00:43.193294+0000","last_became_active":"2024-09-14T06:28:52.775997+0000","last_became_peered":"2024-09-14T06:28:52.775997+0000","last_unstale":"2024-09-14T07:00:43.193294+0000","last_undegraded":"2024-09-14T07:00:43.193294+0000","last_fullsized":"2024-09-14T07:00:43.193294+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:48.639577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000668876,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":892,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192082+0000","last_change":"2024-09-14T06:28:48.763379+0000","last_active":"2024-09-14T07:00:43.192082+0000","last_peered":"2024-09-14T07:00:43.192082+0000","last_clean":"2024-09-14T07:00:43.192082+0000","last_became_active":"2024-09-14T06:28:48.763050+0000","last_became_peered":"2024-09-14T06:28:48.763050+0000","last_unstale":"2024-09-14T07:00:43.192082+0000","last_undegraded":"2024-09-14T07:00:43.192082+0000","last_fullsized":"2024-09-14T07:00:43.192082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_clean_scrub_stamp":"2024-09-14T06:28:47.755018+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:01:18.238865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738757,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838760,"kb_used_data":4756,"kb_used_omap":126,"kb_used_meta":833857,"kb_avail":93533080,"statfs":{"total":96636764160,"available":95777873920,"internally_reserved":0,"allocated":4870144,"data_stored":2281768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":129574,"internal_metadata":853870042},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738757,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957872,"kb_used_data":5340,"kb_used_omap":114,"kb_used_meta":952397,"kb_avail":93413968,"statfs":{"total":96636764160,"available":95655903232,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117715,"internal_metadata":975254573},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1216880,"kb_used_data":5340,"kb_used_omap":135,"kb_used_meta":1211384,"kb_avail":93154960,"statfs":{"total":96636764160,"available":95390679040,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":138887,"internal_metadata":1240457593},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:00:50.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T07:00:51.218 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T07:00:51.219 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-09-14T07:00:51.232 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1370,"stamp":"2024-09-14T07:00:49.956837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3013512,"kb_used_data":15436,"kb_used_omap":377,"kb_used_meta":2997638,"kb_avail":280102008,"statfs":{"total":289910292480,"available":286824456192,"internally_reserved":0,"allocated":15806464,"data_stored":8026078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":386176,"internal_metadata":3069582208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-1963,"num_read_kb":-1952,"num_write":-132,"num_write_kb":-129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.773526"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194187+0000","last_change":"2024-09-14T06:28:54.782642+0000","last_active":"2024-09-14T07:00:43.194187+0000","last_peered":"2024-09-14T07:00:43.194187+0000","last_clean":"2024-09-14T07:00:43.194187+0000","last_became_active":"2024-09-14T06:28:52.776292+0000","last_became_peered":"2024-09-14T06:28:52.776292+0000","last_unstale":"2024-09-14T07:00:43.194187+0000","last_undegraded":"2024-09-14T07:00:43.194187+0000","last_fullsized":"2024-09-14T07:00:43.194187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:48:57.788740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010453089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192722+0000","last_change":"2024-09-14T06:28:54.782290+0000","last_active":"2024-09-14T07:00:43.192722+0000","last_peered":"2024-09-14T07:00:43.192722+0000","last_clean":"2024-09-14T07:00:43.192722+0000","last_became_active":"2024-09-14T06:28:52.776371+0000","last_became_peered":"2024-09-14T06:28:52.776371+0000","last_unstale":"2024-09-14T07:00:43.192722+0000","last_undegraded":"2024-09-14T07:00:43.192722+0000","last_fullsized":"2024-09-14T07:00:43.192722+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:49:06.151379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077233399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194838+0000","last_change":"2024-09-14T06:28:54.782567+0000","last_active":"2024-09-14T07:00:43.194838+0000","last_peered":"2024-09-14T07:00:43.194838+0000","last_clean":"2024-09-14T07:00:43.194838+0000","last_became_active":"2024-09-14T06:28:52.776250+0000","last_became_peered":"2024-09-14T06:28:52.776250+0000","last_unstale":"2024-09-14T07:00:43.194838+0000","last_undegraded":"2024-09-14T07:00:43.194838+0000","last_fullsized":"2024-09-14T07:00:43.194838+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:21:07.454036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011012839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193506+0000","last_change":"2024-09-14T06:28:54.781991+0000","last_active":"2024-09-14T07:00:43.193506+0000","last_peered":"2024-09-14T07:00:43.193506+0000","last_clean":"2024-09-14T07:00:43.193506+0000","last_became_active":"2024-09-14T06:28:52.776740+0000","last_became_peered":"2024-09-14T06:28:52.776740+0000","last_unstale":"2024-09-14T07:00:43.193506+0000","last_undegraded":"2024-09-14T07:00:43.193506+0000","last_fullsized":"2024-09-14T07:00:43.193506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:17:00.207670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064401499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.309808+0000","last_change":"2024-09-14T06:28:54.780145+0000","last_active":"2024-09-14T07:00:43.309808+0000","last_peered":"2024-09-14T07:00:43.309808+0000","last_clean":"2024-09-14T07:00:43.309808+0000","last_became_active":"2024-09-14T06:28:52.776357+0000","last_became_peered":"2024-09-14T06:28:52.776357+0000","last_unstale":"2024-09-14T07:00:43.309808+0000","last_undegraded":"2024-09-14T07:00:43.309808+0000","last_fullsized":"2024-09-14T07:00:43.309808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:57:29.100954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096998400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.402626+0000","last_change":"2024-09-14T06:28:54.777770+0000","last_active":"2024-09-14T07:00:43.402626+0000","last_peered":"2024-09-14T07:00:43.402626+0000","last_clean":"2024-09-14T07:00:43.402626+0000","last_became_active":"2024-09-14T06:28:52.777933+0000","last_became_peered":"2024-09-14T06:28:52.777933+0000","last_unstale":"2024-09-14T07:00:43.402626+0000","last_undegraded":"2024-09-14T07:00:43.402626+0000","last_fullsized":"2024-09-14T07:00:43.402626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:50.657246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049603799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.404434+0000","last_change":"2024-09-14T06:28:54.777772+0000","last_active":"2024-09-14T07:00:43.404434+0000","last_peered":"2024-09-14T07:00:43.404434+0000","last_clean":"2024-09-14T07:00:43.404434+0000","last_became_active":"2024-09-14T06:28:52.777950+0000","last_became_peered":"2024-09-14T06:28:52.777950+0000","last_unstale":"2024-09-14T07:00:43.404434+0000","last_undegraded":"2024-09-14T07:00:43.404434+0000","last_fullsized":"2024-09-14T07:00:43.404434+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:44:17.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504179,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193294+0000","last_change":"2024-09-14T06:28:54.782038+0000","last_active":"2024-09-14T07:00:43.193294+0000","last_peered":"2024-09-14T07:00:43.193294+0000","last_clean":"2024-09-14T07:00:43.193294+0000","last_became_active":"2024-09-14T06:28:52.775997+0000","last_became_peered":"2024-09-14T06:28:52.775997+0000","last_unstale":"2024-09-14T07:00:43.193294+0000","last_undegraded":"2024-09-14T07:00:43.193294+0000","last_fullsized":"2024-09-14T07:00:43.193294+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:48.639577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000668876,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":892,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192082+0000","last_change":"2024-09-14T06:28:48.763379+0000","last_active":"2024-09-14T07:00:43.192082+0000","last_peered":"2024-09-14T07:00:43.192082+0000","last_clean":"2024-09-14T07:00:43.192082+0000","last_became_active":"2024-09-14T06:28:48.763050+0000","last_became_peered":"2024-09-14T06:28:48.763050+0000","last_unstale":"2024-09-14T07:00:43.192082+0000","last_undegraded":"2024-09-14T07:00:43.192082+0000","last_fullsized":"2024-09-14T07:00:43.192082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_clean_scrub_stamp":"2024-09-14T06:28:47.755018+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:01:18.238865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738757,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838760,"kb_used_data":4756,"kb_used_omap":126,"kb_used_meta":833857,"kb_avail":93533080,"statfs":{"total":96636764160,"available":95777873920,"internally_reserved":0,"allocated":4870144,"data_stored":2281768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":129574,"internal_metadata":853870042},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738757,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957872,"kb_used_data":5340,"kb_used_omap":114,"kb_used_meta":952397,"kb_avail":93413968,"statfs":{"total":96636764160,"available":95655903232,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117715,"internal_metadata":975254573},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1216880,"kb_used_data":5340,"kb_used_omap":135,"kb_used_meta":1211384,"kb_avail":93154960,"statfs":{"total":96636764160,"available":95390679040,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":138887,"internal_metadata":1240457593},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:00:51.233 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-14T07:00:51.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T07:00:51.512 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T07:00:51.513 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-09-14T07:00:51.524 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1370,"stamp":"2024-09-14T07:00:49.956837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3013512,"kb_used_data":15436,"kb_used_omap":377,"kb_used_meta":2997638,"kb_avail":280102008,"statfs":{"total":289910292480,"available":286824456192,"internally_reserved":0,"allocated":15806464,"data_stored":8026078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":386176,"internal_metadata":3069582208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-1963,"num_read_kb":-1952,"num_write":-132,"num_write_kb":-129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.773526"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194187+0000","last_change":"2024-09-14T06:28:54.782642+0000","last_active":"2024-09-14T07:00:43.194187+0000","last_peered":"2024-09-14T07:00:43.194187+0000","last_clean":"2024-09-14T07:00:43.194187+0000","last_became_active":"2024-09-14T06:28:52.776292+0000","last_became_peered":"2024-09-14T06:28:52.776292+0000","last_unstale":"2024-09-14T07:00:43.194187+0000","last_undegraded":"2024-09-14T07:00:43.194187+0000","last_fullsized":"2024-09-14T07:00:43.194187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:48:57.788740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010453089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192722+0000","last_change":"2024-09-14T06:28:54.782290+0000","last_active":"2024-09-14T07:00:43.192722+0000","last_peered":"2024-09-14T07:00:43.192722+0000","last_clean":"2024-09-14T07:00:43.192722+0000","last_became_active":"2024-09-14T06:28:52.776371+0000","last_became_peered":"2024-09-14T06:28:52.776371+0000","last_unstale":"2024-09-14T07:00:43.192722+0000","last_undegraded":"2024-09-14T07:00:43.192722+0000","last_fullsized":"2024-09-14T07:00:43.192722+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:49:06.151379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077233399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194838+0000","last_change":"2024-09-14T06:28:54.782567+0000","last_active":"2024-09-14T07:00:43.194838+0000","last_peered":"2024-09-14T07:00:43.194838+0000","last_clean":"2024-09-14T07:00:43.194838+0000","last_became_active":"2024-09-14T06:28:52.776250+0000","last_became_peered":"2024-09-14T06:28:52.776250+0000","last_unstale":"2024-09-14T07:00:43.194838+0000","last_undegraded":"2024-09-14T07:00:43.194838+0000","last_fullsized":"2024-09-14T07:00:43.194838+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:21:07.454036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011012839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193506+0000","last_change":"2024-09-14T06:28:54.781991+0000","last_active":"2024-09-14T07:00:43.193506+0000","last_peered":"2024-09-14T07:00:43.193506+0000","last_clean":"2024-09-14T07:00:43.193506+0000","last_became_active":"2024-09-14T06:28:52.776740+0000","last_became_peered":"2024-09-14T06:28:52.776740+0000","last_unstale":"2024-09-14T07:00:43.193506+0000","last_undegraded":"2024-09-14T07:00:43.193506+0000","last_fullsized":"2024-09-14T07:00:43.193506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:17:00.207670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064401499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.309808+0000","last_change":"2024-09-14T06:28:54.780145+0000","last_active":"2024-09-14T07:00:43.309808+0000","last_peered":"2024-09-14T07:00:43.309808+0000","last_clean":"2024-09-14T07:00:43.309808+0000","last_became_active":"2024-09-14T06:28:52.776357+0000","last_became_peered":"2024-09-14T06:28:52.776357+0000","last_unstale":"2024-09-14T07:00:43.309808+0000","last_undegraded":"2024-09-14T07:00:43.309808+0000","last_fullsized":"2024-09-14T07:00:43.309808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:57:29.100954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096998400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.402626+0000","last_change":"2024-09-14T06:28:54.777770+0000","last_active":"2024-09-14T07:00:43.402626+0000","last_peered":"2024-09-14T07:00:43.402626+0000","last_clean":"2024-09-14T07:00:43.402626+0000","last_became_active":"2024-09-14T06:28:52.777933+0000","last_became_peered":"2024-09-14T06:28:52.777933+0000","last_unstale":"2024-09-14T07:00:43.402626+0000","last_undegraded":"2024-09-14T07:00:43.402626+0000","last_fullsized":"2024-09-14T07:00:43.402626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:50.657246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049603799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.404434+0000","last_change":"2024-09-14T06:28:54.777772+0000","last_active":"2024-09-14T07:00:43.404434+0000","last_peered":"2024-09-14T07:00:43.404434+0000","last_clean":"2024-09-14T07:00:43.404434+0000","last_became_active":"2024-09-14T06:28:52.777950+0000","last_became_peered":"2024-09-14T06:28:52.777950+0000","last_unstale":"2024-09-14T07:00:43.404434+0000","last_undegraded":"2024-09-14T07:00:43.404434+0000","last_fullsized":"2024-09-14T07:00:43.404434+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:44:17.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504179,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193294+0000","last_change":"2024-09-14T06:28:54.782038+0000","last_active":"2024-09-14T07:00:43.193294+0000","last_peered":"2024-09-14T07:00:43.193294+0000","last_clean":"2024-09-14T07:00:43.193294+0000","last_became_active":"2024-09-14T06:28:52.775997+0000","last_became_peered":"2024-09-14T06:28:52.775997+0000","last_unstale":"2024-09-14T07:00:43.193294+0000","last_undegraded":"2024-09-14T07:00:43.193294+0000","last_fullsized":"2024-09-14T07:00:43.193294+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:48.639577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000668876,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":892,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192082+0000","last_change":"2024-09-14T06:28:48.763379+0000","last_active":"2024-09-14T07:00:43.192082+0000","last_peered":"2024-09-14T07:00:43.192082+0000","last_clean":"2024-09-14T07:00:43.192082+0000","last_became_active":"2024-09-14T06:28:48.763050+0000","last_became_peered":"2024-09-14T06:28:48.763050+0000","last_unstale":"2024-09-14T07:00:43.192082+0000","last_undegraded":"2024-09-14T07:00:43.192082+0000","last_fullsized":"2024-09-14T07:00:43.192082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_clean_scrub_stamp":"2024-09-14T06:28:47.755018+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:01:18.238865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738757,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838760,"kb_used_data":4756,"kb_used_omap":126,"kb_used_meta":833857,"kb_avail":93533080,"statfs":{"total":96636764160,"available":95777873920,"internally_reserved":0,"allocated":4870144,"data_stored":2281768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":129574,"internal_metadata":853870042},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738757,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957872,"kb_used_data":5340,"kb_used_omap":114,"kb_used_meta":952397,"kb_avail":93413968,"statfs":{"total":96636764160,"available":95655903232,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117715,"internal_metadata":975254573},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1216880,"kb_used_data":5340,"kb_used_omap":135,"kb_used_meta":1211384,"kb_avail":93154960,"statfs":{"total":96636764160,"available":95390679040,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":138887,"internal_metadata":1240457593},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:00:51.525 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T07:00:51.809 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T07:00:51.809 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":411,"fsid":"a7356329-2cae-44e1-a06f-290f5fd22015","created":"2024-09-14T06:28:35.900384+0000","modified":"2024-09-14T07:00:43.185199+0000","last_up_change":"2024-09-14T06:28:45.753509+0000","last_in_change":"2024-09-14T06:28:36.735545+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:47.496517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T06:28:50.986399+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7de0430d-00ac-4a23-b678-2d3db13c7c08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6801","nonce":4002621419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6803","nonce":4002621419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6807","nonce":4002621419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":4002621419},{"type":"v1","addr":"172.21.15.170:6805","nonce":4002621419}]},"public_addr":"172.21.15.170:6801/4002621419","cluster_addr":"172.21.15.170:6803/4002621419","heartbeat_back_addr":"172.21.15.170:6807/4002621419","heartbeat_front_addr":"172.21.15.170:6805/4002621419","state":["exists","up"]},{"osd":1,"uuid":"2ae5c1c1-21e0-47aa-ba83-adc731863719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6809","nonce":3258460295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6811","nonce":3258460295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6815","nonce":3258460295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":3258460295},{"type":"v1","addr":"172.21.15.170:6813","nonce":3258460295}]},"public_addr":"172.21.15.170:6809/3258460295","cluster_addr":"172.21.15.170:6811/3258460295","heartbeat_back_addr":"172.21.15.170:6815/3258460295","heartbeat_front_addr":"172.21.15.170:6813/3258460295","state":["exists","up"]},{"osd":2,"uuid":"eab8881a-6d1e-495c-bba6-27c1ac2b98d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6817","nonce":2319540746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6819","nonce":2319540746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6823","nonce":2319540746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":2319540746},{"type":"v1","addr":"172.21.15.170:6821","nonce":2319540746}]},"public_addr":"172.21.15.170:6817/2319540746","cluster_addr":"172.21.15.170:6819/2319540746","heartbeat_back_addr":"172.21.15.170:6823/2319540746","heartbeat_front_addr":"172.21.15.170:6821/2319540746","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.007661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:42.984623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:43.008465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.170:0/2254682370":"2024-09-14T08:00:30.954382+0000","172.21.15.170:0/3739395690":"2024-09-14T07:59:54.657953+0000","172.21.15.170:0/940589038":"2024-09-14T07:59:11.315643+0000","172.21.15.170:0/1543209806":"2024-09-14T07:57:54.199596+0000","172.21.15.170:0/920944073":"2024-09-14T07:56:29.793473+0000","172.21.15.170:0/2071894304":"2024-09-14T07:55:45.855377+0000","172.21.15.170:0/363795428":"2024-09-14T07:55:02.092392+0000","172.21.15.170:0/2218960426":"2024-09-14T07:48:56.458345+0000","172.21.15.170:0/2919551905":"2024-09-14T07:49:07.889213+0000","172.21.15.170:0/3861130040":"2024-09-14T07:50:44.476696+0000","172.21.15.170:0/4162627728":"2024-09-14T07:48:29.006445+0000","172.21.15.170:0/2604051990":"2024-09-14T07:49:02.521978+0000","172.21.15.170:0/2134931760":"2024-09-14T07:31:58.728311+0000","172.21.15.170:0/4195680358":"2024-09-14T07:48:15.820608+0000","172.21.15.170:0/3351568299":"2024-09-14T07:43:25.133488+0000","172.21.15.170:0/3542034889":"2024-09-14T07:48:46.570268+0000","172.21.15.170:0/2029793399":"2024-09-14T07:52:14.269352+0000","172.21.15.170:0/1044590568":"2024-09-14T07:54:16.544326+0000","172.21.15.170:0/592467440":"2024-09-14T07:50:39.245215+0000","172.21.15.170:0/3486588897":"2024-09-14T07:47:14.413825+0000","172.21.15.170:0/2658274836":"2024-09-14T07:51:15.475489+0000","172.21.15.170:0/2578051548":"2024-09-14T07:48:07.897122+0000","172.21.15.170:0/996463600":"2024-09-14T07:33:31.477248+0000","172.21.15.170:0/3071882208":"2024-09-14T07:51:30.364228+0000","172.21.15.170:0/3579264602":"2024-09-14T07:45:49.873529+0000","172.21.15.170:0/3031351743":"2024-09-14T07:47:31.119202+0000","172.21.15.170:0/4165441957":"2024-09-14T07:36:40.092750+0000","172.21.15.170:0/711013294":"2024-09-14T07:48:40.756635+0000","172.21.15.170:0/101141012":"2024-09-14T07:48:23.488711+0000","172.21.15.170:0/3619156484":"2024-09-14T07:35:49.561337+0000","172.21.15.170:0/2315213199":"2024-09-14T07:51:55.311314+0000","172.21.15.170:0/777193507":"2024-09-14T07:31:07.591767+0000","172.21.15.170:0/1231182199":"2024-09-14T07:45:34.187745+0000","172.21.15.170:0/2912640664":"2024-09-14T07:42:07.726921+0000","172.21.15.170:0/3931963306":"2024-09-14T07:35:04.844713+0000","172.21.15.170:0/194293609":"2024-09-14T07:37:35.436573+0000","172.21.15.170:0/3843898172":"2024-09-14T07:41:23.239849+0000","172.21.15.170:0/3946679538":"2024-09-14T07:52:20.339022+0000","172.21.15.170:0/921918992":"2024-09-14T07:49:45.278335+0000","172.21.15.170:0/1035662076":"2024-09-14T07:50:12.345939+0000","172.21.15.170:0/264234311":"2024-09-14T07:57:10.406876+0000","172.21.15.170:0/3602318976":"2024-09-14T07:46:31.061076+0000","172.21.15.170:0/3827063477":"2024-09-14T07:49:14.412267+0000","172.21.15.170:0/527211269":"2024-09-14T07:49:27.912361+0000","172.21.15.170:0/1804093208":"2024-09-14T07:58:35.463296+0000","172.21.15.170:0/3140111513":"2024-09-14T07:34:21.307845+0000","172.21.15.170:0/3110979877":"2024-09-14T07:49:33.055815+0000","172.21.15.170:0/2378572352":"2024-09-14T07:49:38.262091+0000","172.21.15.170:0/2989734206":"2024-09-14T07:52:30.258975+0000","172.21.15.170:0/3814286467":"2024-09-14T07:51:45.222234+0000","172.21.15.170:0/3831747099":"2024-09-14T07:49:51.303898+0000","172.21.15.170:0/1725838906":"2024-09-14T07:49:57.111597+0000","172.21.15.170:0/2171312302":"2024-09-14T07:50:17.099360+0000","172.21.15.170:0/2545632916":"2024-09-14T07:51:25.488120+0000","172.21.15.170:0/1305857377":"2024-09-14T07:50:06.632689+0000","172.21.15.170:0/3322120812":"2024-09-14T07:50:22.390104+0000","172.21.15.170:0/2611310381":"2024-09-14T07:32:44.724263+0000","172.21.15.170:0/1004668840":"2024-09-14T07:42:07.381859+0000","172.21.15.170:0/1896918188":"2024-09-14T07:50:27.473706+0000","172.21.15.170:0/1564287313":"2024-09-14T07:48:35.239850+0000","172.21.15.170:0/3903049408":"2024-09-14T07:50:32.375047+0000","172.21.15.170:0/4246586819":"2024-09-14T07:38:22.585022+0000","172.21.15.170:0/316033375":"2024-09-14T07:50:54.515117+0000","172.21.15.170:0/3422983602":"2024-09-14T07:51:00.848683+0000","172.21.15.170:0/397447591":"2024-09-14T07:51:36.482499+0000","172.21.15.170:0/1476560167":"2024-09-14T07:51:06.494634+0000","172.21.15.170:0/944480121":"2024-09-14T07:51:50.515069+0000","172.21.15.170:0/2485436429":"2024-09-14T07:52:05.855205+0000","172.21.15.170:0/2826520082":"2024-09-14T07:50:49.217897+0000","172.21.15.170:0/3990914408":"2024-09-14T07:52:25.864424+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T07:00:52.824 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-14T07:00:52.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-14T07:00:52.959 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-09-14T07:00:52.959 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T07:00:52.959 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-09-14T07:00:52.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-14T07:00:53.255 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 0 to deep-scrub 2024-09-14T07:00:53.268 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-14T07:00:53.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-14T07:00:53.391 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-09-14T07:00:53.391 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T07:00:53.391 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-09-14T07:00:53.405 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-14T07:00:53.690 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 1 to deep-scrub 2024-09-14T07:00:53.704 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-14T07:00:53.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-14T07:00:53.826 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-09-14T07:00:53.826 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T07:00:53.826 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-09-14T07:00:53.837 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-14T07:00:54.121 INFO:teuthology.orchestra.run.smithi170.stderr:instructed osd(s) 2 to deep-scrub 2024-09-14T07:00:54.134 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T07:00:54.415 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T07:00:54.415 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-09-14T07:00:54.429 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1372,"stamp":"2024-09-14T07:00:53.957587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3013512,"kb_used_data":15436,"kb_used_omap":377,"kb_used_meta":2997638,"kb_avail":280102008,"statfs":{"total":289910292480,"available":286824456192,"internally_reserved":0,"allocated":15806464,"data_stored":8026078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":386176,"internal_metadata":3069582208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.769401"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194187+0000","last_change":"2024-09-14T06:28:54.782642+0000","last_active":"2024-09-14T07:00:43.194187+0000","last_peered":"2024-09-14T07:00:43.194187+0000","last_clean":"2024-09-14T07:00:43.194187+0000","last_became_active":"2024-09-14T06:28:52.776292+0000","last_became_peered":"2024-09-14T06:28:52.776292+0000","last_unstale":"2024-09-14T07:00:43.194187+0000","last_undegraded":"2024-09-14T07:00:43.194187+0000","last_fullsized":"2024-09-14T07:00:43.194187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:48:57.788740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010453089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":823,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192722+0000","last_change":"2024-09-14T06:28:54.782290+0000","last_active":"2024-09-14T07:00:43.192722+0000","last_peered":"2024-09-14T07:00:43.192722+0000","last_clean":"2024-09-14T07:00:43.192722+0000","last_became_active":"2024-09-14T06:28:52.776371+0000","last_became_peered":"2024-09-14T06:28:52.776371+0000","last_unstale":"2024-09-14T07:00:43.192722+0000","last_undegraded":"2024-09-14T07:00:43.192722+0000","last_fullsized":"2024-09-14T07:00:43.192722+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:49:06.151379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077233399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.194838+0000","last_change":"2024-09-14T06:28:54.782567+0000","last_active":"2024-09-14T07:00:43.194838+0000","last_peered":"2024-09-14T07:00:43.194838+0000","last_clean":"2024-09-14T07:00:43.194838+0000","last_became_active":"2024-09-14T06:28:52.776250+0000","last_became_peered":"2024-09-14T06:28:52.776250+0000","last_unstale":"2024-09-14T07:00:43.194838+0000","last_undegraded":"2024-09-14T07:00:43.194838+0000","last_fullsized":"2024-09-14T07:00:43.194838+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:21:07.454036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011012839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193506+0000","last_change":"2024-09-14T06:28:54.781991+0000","last_active":"2024-09-14T07:00:43.193506+0000","last_peered":"2024-09-14T07:00:43.193506+0000","last_clean":"2024-09-14T07:00:43.193506+0000","last_became_active":"2024-09-14T06:28:52.776740+0000","last_became_peered":"2024-09-14T06:28:52.776740+0000","last_unstale":"2024-09-14T07:00:43.193506+0000","last_undegraded":"2024-09-14T07:00:43.193506+0000","last_fullsized":"2024-09-14T07:00:43.193506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:17:00.207670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064401499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"411'3","reported_seq":835,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:53.549394+0000","last_change":"2024-09-14T07:00:53.549394+0000","last_active":"2024-09-14T07:00:53.549394+0000","last_peered":"2024-09-14T07:00:53.549394+0000","last_clean":"2024-09-14T07:00:53.549394+0000","last_became_active":"2024-09-14T06:28:52.776357+0000","last_became_peered":"2024-09-14T06:28:52.776357+0000","last_unstale":"2024-09-14T07:00:53.549394+0000","last_undegraded":"2024-09-14T07:00:53.549394+0000","last_fullsized":"2024-09-14T07:00:53.549394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'3","last_scrub_stamp":"2024-09-14T07:00:53.549305+0000","last_deep_scrub":"411'3","last_deep_scrub_stamp":"2024-09-14T07:00:53.549305+0000","last_clean_scrub_stamp":"2024-09-14T07:00:53.549305+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:01:39.837894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096998400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.402626+0000","last_change":"2024-09-14T06:28:54.777770+0000","last_active":"2024-09-14T07:00:43.402626+0000","last_peered":"2024-09-14T07:00:43.402626+0000","last_clean":"2024-09-14T07:00:43.402626+0000","last_became_active":"2024-09-14T06:28:52.777933+0000","last_became_peered":"2024-09-14T06:28:52.777933+0000","last_unstale":"2024-09-14T07:00:43.402626+0000","last_undegraded":"2024-09-14T07:00:43.402626+0000","last_fullsized":"2024-09-14T07:00:43.402626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:50.657246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049603799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.404434+0000","last_change":"2024-09-14T06:28:54.777772+0000","last_active":"2024-09-14T07:00:43.404434+0000","last_peered":"2024-09-14T07:00:43.404434+0000","last_clean":"2024-09-14T07:00:43.404434+0000","last_became_active":"2024-09-14T06:28:52.777950+0000","last_became_peered":"2024-09-14T06:28:52.777950+0000","last_unstale":"2024-09-14T07:00:43.404434+0000","last_undegraded":"2024-09-14T07:00:43.404434+0000","last_fullsized":"2024-09-14T07:00:43.404434+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:44:17.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504179,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":825,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.193294+0000","last_change":"2024-09-14T06:28:54.782038+0000","last_active":"2024-09-14T07:00:43.193294+0000","last_peered":"2024-09-14T07:00:43.193294+0000","last_clean":"2024-09-14T07:00:43.193294+0000","last_became_active":"2024-09-14T06:28:52.775997+0000","last_became_peered":"2024-09-14T06:28:52.775997+0000","last_unstale":"2024-09-14T07:00:43.193294+0000","last_undegraded":"2024-09-14T07:00:43.193294+0000","last_fullsized":"2024-09-14T07:00:43.193294+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:51.764439+0000","last_clean_scrub_stamp":"2024-09-14T06:28:51.764439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:48.639577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000668876,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":892,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.192082+0000","last_change":"2024-09-14T06:28:48.763379+0000","last_active":"2024-09-14T07:00:43.192082+0000","last_peered":"2024-09-14T07:00:43.192082+0000","last_clean":"2024-09-14T07:00:43.192082+0000","last_became_active":"2024-09-14T06:28:48.763050+0000","last_became_peered":"2024-09-14T06:28:48.763050+0000","last_unstale":"2024-09-14T07:00:43.192082+0000","last_undegraded":"2024-09-14T07:00:43.192082+0000","last_fullsized":"2024-09-14T07:00:43.192082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:47.755018+0000","last_clean_scrub_stamp":"2024-09-14T06:28:47.755018+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:01:18.238865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838760,"kb_used_data":4756,"kb_used_omap":126,"kb_used_meta":833857,"kb_avail":93533080,"statfs":{"total":96636764160,"available":95777873920,"internally_reserved":0,"allocated":4870144,"data_stored":2281768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":129574,"internal_metadata":853870042},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738758,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957872,"kb_used_data":5340,"kb_used_omap":114,"kb_used_meta":952397,"kb_avail":93413968,"statfs":{"total":96636764160,"available":95655903232,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117715,"internal_metadata":975254573},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1216880,"kb_used_data":5340,"kb_used_omap":135,"kb_used_meta":1211384,"kb_avail":93154960,"statfs":{"total":96636764160,"available":95390679040,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":138887,"internal_metadata":1240457593},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:00:54.433 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-14T06:28:51.764439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.433 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-14T06:28:51.764439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.433 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-14T06:28:51.764439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.434 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-14T06:28:51.764439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.434 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-14T06:28:51.764439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.434 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-14T06:28:51.764439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.434 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-14T06:28:51.764439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.434 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-14T06:28:47.755018+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=47, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=0, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:00:54.434 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-14T07:01:14.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T07:01:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-09-14T07:01:14.729 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-09-14T07:01:14.740 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":1382,"stamp":"2024-09-14T07:01:13.962150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3013500,"kb_used_data":15436,"kb_used_omap":377,"kb_used_meta":2997638,"kb_avail":280102020,"statfs":{"total":289910292480,"available":286824468480,"internally_reserved":0,"allocated":15806464,"data_stored":8026078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":386176,"internal_metadata":3069582208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":831,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:58.839256+0000","last_change":"2024-09-14T07:00:58.839256+0000","last_active":"2024-09-14T07:00:58.839256+0000","last_peered":"2024-09-14T07:00:58.839256+0000","last_clean":"2024-09-14T07:00:58.839256+0000","last_became_active":"2024-09-14T06:28:52.776292+0000","last_became_peered":"2024-09-14T06:28:52.776292+0000","last_unstale":"2024-09-14T07:00:58.839256+0000","last_undegraded":"2024-09-14T07:00:58.839256+0000","last_fullsized":"2024-09-14T07:00:58.839256+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:00:58.839170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:00:58.839170+0000","last_clean_scrub_stamp":"2024-09-14T07:00:58.839170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T17:29:54.795238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010453089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":831,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:56.812167+0000","last_change":"2024-09-14T07:00:56.812167+0000","last_active":"2024-09-14T07:00:56.812167+0000","last_peered":"2024-09-14T07:00:56.812167+0000","last_clean":"2024-09-14T07:00:56.812167+0000","last_became_active":"2024-09-14T06:28:52.776371+0000","last_became_peered":"2024-09-14T06:28:52.776371+0000","last_unstale":"2024-09-14T07:00:56.812167+0000","last_undegraded":"2024-09-14T07:00:56.812167+0000","last_fullsized":"2024-09-14T07:00:56.812167+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:00:56.812064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:00:56.812064+0000","last_clean_scrub_stamp":"2024-09-14T07:00:56.812064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:15:23.869522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077233399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":832,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:57.806901+0000","last_change":"2024-09-14T07:00:57.806901+0000","last_active":"2024-09-14T07:00:57.806901+0000","last_peered":"2024-09-14T07:00:57.806901+0000","last_clean":"2024-09-14T07:00:57.806901+0000","last_became_active":"2024-09-14T06:28:52.776250+0000","last_became_peered":"2024-09-14T06:28:52.776250+0000","last_unstale":"2024-09-14T07:00:57.806901+0000","last_undegraded":"2024-09-14T07:00:57.806901+0000","last_fullsized":"2024-09-14T07:00:57.806901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:00:57.806804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:00:57.806804+0000","last_clean_scrub_stamp":"2024-09-14T07:00:57.806804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:39:43.656268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011012839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":832,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:53.820083+0000","last_change":"2024-09-14T07:00:53.820083+0000","last_active":"2024-09-14T07:00:53.820083+0000","last_peered":"2024-09-14T07:00:53.820083+0000","last_clean":"2024-09-14T07:00:53.820083+0000","last_became_active":"2024-09-14T06:28:52.776740+0000","last_became_peered":"2024-09-14T06:28:52.776740+0000","last_unstale":"2024-09-14T07:00:53.820083+0000","last_undegraded":"2024-09-14T07:00:53.820083+0000","last_fullsized":"2024-09-14T07:00:53.820083+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:00:53.820012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:00:53.820012+0000","last_clean_scrub_stamp":"2024-09-14T07:00:53.820012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:09:00.699196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064401499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"411'3","reported_seq":835,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:53.549394+0000","last_change":"2024-09-14T07:00:53.549394+0000","last_active":"2024-09-14T07:00:53.549394+0000","last_peered":"2024-09-14T07:00:53.549394+0000","last_clean":"2024-09-14T07:00:53.549394+0000","last_became_active":"2024-09-14T06:28:52.776357+0000","last_became_peered":"2024-09-14T06:28:52.776357+0000","last_unstale":"2024-09-14T07:00:53.549394+0000","last_undegraded":"2024-09-14T07:00:53.549394+0000","last_fullsized":"2024-09-14T07:00:53.549394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'3","last_scrub_stamp":"2024-09-14T07:00:53.549305+0000","last_deep_scrub":"411'3","last_deep_scrub_stamp":"2024-09-14T07:00:53.549305+0000","last_clean_scrub_stamp":"2024-09-14T07:00:53.549305+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:01:39.837894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096998400000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":833,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:54.378900+0000","last_change":"2024-09-14T07:00:54.378900+0000","last_active":"2024-09-14T07:00:54.378900+0000","last_peered":"2024-09-14T07:00:54.378900+0000","last_clean":"2024-09-14T07:00:54.378900+0000","last_became_active":"2024-09-14T06:28:52.777933+0000","last_became_peered":"2024-09-14T06:28:52.777933+0000","last_unstale":"2024-09-14T07:00:54.378900+0000","last_undegraded":"2024-09-14T07:00:54.378900+0000","last_fullsized":"2024-09-14T07:00:54.378900+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:00:54.378825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:00:54.378825+0000","last_clean_scrub_stamp":"2024-09-14T07:00:54.378825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T18:30:45.849990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049603799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":834,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:55.407940+0000","last_change":"2024-09-14T07:00:55.407876+0000","last_active":"2024-09-14T07:00:55.407940+0000","last_peered":"2024-09-14T07:00:55.407940+0000","last_clean":"2024-09-14T07:00:55.407940+0000","last_became_active":"2024-09-14T06:28:52.777950+0000","last_became_peered":"2024-09-14T06:28:52.777950+0000","last_unstale":"2024-09-14T07:00:55.407940+0000","last_undegraded":"2024-09-14T07:00:55.407940+0000","last_fullsized":"2024-09-14T07:00:55.407940+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:00:55.407799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:00:55.407799+0000","last_clean_scrub_stamp":"2024-09-14T07:00:55.407799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:45:27.296909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504179,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"411'2","reported_seq":834,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:55.808867+0000","last_change":"2024-09-14T07:00:55.808867+0000","last_active":"2024-09-14T07:00:55.808867+0000","last_peered":"2024-09-14T07:00:55.808867+0000","last_clean":"2024-09-14T07:00:55.808867+0000","last_became_active":"2024-09-14T06:28:52.775997+0000","last_became_peered":"2024-09-14T06:28:52.775997+0000","last_unstale":"2024-09-14T07:00:55.808867+0000","last_undegraded":"2024-09-14T07:00:55.808867+0000","last_fullsized":"2024-09-14T07:00:55.808867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'2","last_scrub_stamp":"2024-09-14T07:00:55.808788+0000","last_deep_scrub":"411'2","last_deep_scrub_stamp":"2024-09-14T07:00:55.808788+0000","last_clean_scrub_stamp":"2024-09-14T07:00:55.808788+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T17:37:52.358709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000668876,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"411'54","reported_seq":903,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-14T07:00:54.822712+0000","last_change":"2024-09-14T07:00:54.822630+0000","last_active":"2024-09-14T07:00:54.822712+0000","last_peered":"2024-09-14T07:00:54.822712+0000","last_clean":"2024-09-14T07:00:54.822712+0000","last_became_active":"2024-09-14T06:28:48.763050+0000","last_became_peered":"2024-09-14T06:28:48.763050+0000","last_unstale":"2024-09-14T07:00:54.822712+0000","last_undegraded":"2024-09-14T07:00:54.822712+0000","last_fullsized":"2024-09-14T07:00:54.822712+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'54","last_scrub_stamp":"2024-09-14T07:00:54.822473+0000","last_deep_scrub":"411'54","last_deep_scrub_stamp":"2024-09-14T07:00:54.822473+0000","last_clean_scrub_stamp":"2024-09-14T07:00:54.822473+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T18:22:02.983118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738762,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838756,"kb_used_data":4756,"kb_used_omap":126,"kb_used_meta":833857,"kb_avail":93533084,"statfs":{"total":96636764160,"available":95777878016,"internally_reserved":0,"allocated":4870144,"data_stored":2281768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":129574,"internal_metadata":853870042},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738762,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957868,"kb_used_data":5340,"kb_used_omap":114,"kb_used_meta":952397,"kb_avail":93413972,"statfs":{"total":96636764160,"available":95655907328,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117715,"internal_metadata":975254573},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738762,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1216876,"kb_used_data":5340,"kb_used_omap":135,"kb_used_meta":1211384,"kb_avail":93154964,"statfs":{"total":96636764160,"available":95390683136,"internally_reserved":0,"allocated":5468160,"data_stored":2872155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":138887,"internal_metadata":1240457593},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:01:14.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-14T07:01:15.043 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-14T07:01:15.043 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-14T07:01:15.044 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-14T07:01:15.044 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:15.117 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-14T07:01:15.118 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-14T07:01:15.118 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:15.196 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-14T07:01:15.197 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-14T07:01:15.197 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:15.268 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-14T07:01:15.268 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-14T07:01:15.269 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-14T07:01:15.269 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:15.300 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-14T07:01:15.301 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-14T07:01:15.301 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-14T07:01:15.301 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:15.314 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-14T07:01:15.314 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-14T07:01:15.314 DEBUG:teuthology.orchestra.run.smithi170:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T07:01:15.428 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T07:01:15.429 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-14T07:01:15.672 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T07:01:15.672 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-14T07:01:15.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T07:01:15.802 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-14T07:01:15.927 INFO:tasks.ceph:Archiving mon data... 2024-09-14T07:01:15.929 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/data/cluster2.mon.a.tgz 2024-09-14T07:01:15.929 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-09-14T07:01:15.944 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.yWNwtUgNHG 2024-09-14T07:01:15.944 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.yWNwtUgNHG 2024-09-14T07:01:16.118 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.yWNwtUgNHG 2024-09-14T07:01:16.207 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.yWNwtUgNHG is 502KB 2024-09-14T07:01:16.305 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.yWNwtUgNHG 2024-09-14T07:01:16.328 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-14T07:01:16.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T07:01:16.331 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T07:01:16.493 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-14T07:01:16.503 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-14T07:01:16.503 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T07:01:17.081 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T07:01:17.082 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-14T07:01:17.094 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1335,"stamp":"2024-09-14T07:01:16.456069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4787172,"kb_used_data":9448,"kb_used_omap":435,"kb_used_meta":4777228,"kb_avail":278328348,"statfs":{"total":289910292480,"available":285008228352,"internally_reserved":0,"allocated":9674752,"data_stored":5652238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":445520,"internal_metadata":4891882416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002343"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083819+0000","last_change":"2024-09-14T06:28:16.071845+0000","last_active":"2024-09-14T07:00:43.083819+0000","last_peered":"2024-09-14T07:00:43.083819+0000","last_clean":"2024-09-14T07:00:43.083819+0000","last_became_active":"2024-09-14T06:28:14.066720+0000","last_became_peered":"2024-09-14T06:28:14.066720+0000","last_unstale":"2024-09-14T07:00:43.083819+0000","last_undegraded":"2024-09-14T07:00:43.083819+0000","last_fullsized":"2024-09-14T07:00:43.083819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:18.865038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078953200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085739+0000","last_change":"2024-09-14T06:28:16.072384+0000","last_active":"2024-09-14T07:00:43.085739+0000","last_peered":"2024-09-14T07:00:43.085739+0000","last_clean":"2024-09-14T07:00:43.085739+0000","last_became_active":"2024-09-14T06:28:14.067568+0000","last_became_peered":"2024-09-14T06:28:14.067568+0000","last_unstale":"2024-09-14T07:00:43.085739+0000","last_undegraded":"2024-09-14T07:00:43.085739+0000","last_fullsized":"2024-09-14T07:00:43.085739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:25:25.337512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.087088+0000","last_change":"2024-09-14T06:28:16.072011+0000","last_active":"2024-09-14T07:00:43.087088+0000","last_peered":"2024-09-14T07:00:43.087088+0000","last_clean":"2024-09-14T07:00:43.087088+0000","last_became_active":"2024-09-14T06:28:14.065921+0000","last_became_peered":"2024-09-14T06:28:14.065921+0000","last_unstale":"2024-09-14T07:00:43.087088+0000","last_undegraded":"2024-09-14T07:00:43.087088+0000","last_fullsized":"2024-09-14T07:00:43.087088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:45:01.691891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092469000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085184+0000","last_change":"2024-09-14T06:28:16.072360+0000","last_active":"2024-09-14T07:00:43.085184+0000","last_peered":"2024-09-14T07:00:43.085184+0000","last_clean":"2024-09-14T07:00:43.085184+0000","last_became_active":"2024-09-14T06:28:14.067522+0000","last_became_peered":"2024-09-14T06:28:14.067522+0000","last_unstale":"2024-09-14T07:00:43.085184+0000","last_undegraded":"2024-09-14T07:00:43.085184+0000","last_fullsized":"2024-09-14T07:00:43.085184+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:56.670133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056764,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":693,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:42.752297+0000","last_change":"2024-09-14T06:28:16.069659+0000","last_active":"2024-09-14T07:00:42.752297+0000","last_peered":"2024-09-14T07:00:42.752297+0000","last_clean":"2024-09-14T07:00:42.752297+0000","last_became_active":"2024-09-14T06:28:14.065736+0000","last_became_peered":"2024-09-14T06:28:14.065736+0000","last_unstale":"2024-09-14T07:00:42.752297+0000","last_undegraded":"2024-09-14T07:00:42.752297+0000","last_fullsized":"2024-09-14T07:00:42.752297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:29.974020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010548370000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345473+0000","last_change":"2024-09-14T06:28:16.066424+0000","last_active":"2024-09-14T07:00:43.345473+0000","last_peered":"2024-09-14T07:00:43.345473+0000","last_clean":"2024-09-14T07:00:43.345473+0000","last_became_active":"2024-09-14T06:28:14.065677+0000","last_became_peered":"2024-09-14T06:28:14.065677+0000","last_unstale":"2024-09-14T07:00:43.345473+0000","last_undegraded":"2024-09-14T07:00:43.345473+0000","last_fullsized":"2024-09-14T07:00:43.345473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:36.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047781599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345573+0000","last_change":"2024-09-14T06:28:16.066327+0000","last_active":"2024-09-14T07:00:43.345573+0000","last_peered":"2024-09-14T07:00:43.345573+0000","last_clean":"2024-09-14T07:00:43.345573+0000","last_became_active":"2024-09-14T06:28:14.067147+0000","last_became_peered":"2024-09-14T06:28:14.067147+0000","last_unstale":"2024-09-14T07:00:43.345573+0000","last_undegraded":"2024-09-14T07:00:43.345573+0000","last_fullsized":"2024-09-14T07:00:43.345573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:19:13.309078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042511200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":692,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083496+0000","last_change":"2024-09-14T06:28:16.071663+0000","last_active":"2024-09-14T07:00:43.083496+0000","last_peered":"2024-09-14T07:00:43.083496+0000","last_clean":"2024-09-14T07:00:43.083496+0000","last_became_active":"2024-09-14T06:28:14.065753+0000","last_became_peered":"2024-09-14T06:28:14.065753+0000","last_unstale":"2024-09-14T07:00:43.083496+0000","last_undegraded":"2024-09-14T07:00:43.083496+0000","last_fullsized":"2024-09-14T07:00:43.083496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:38:56.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066619800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":760,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.086573+0000","last_change":"2024-09-14T06:28:10.140562+0000","last_active":"2024-09-14T07:00:43.086573+0000","last_peered":"2024-09-14T07:00:43.086573+0000","last_clean":"2024-09-14T07:00:43.086573+0000","last_became_active":"2024-09-14T06:28:10.139552+0000","last_became_peered":"2024-09-14T06:28:10.139552+0000","last_unstale":"2024-09-14T07:00:43.086573+0000","last_undegraded":"2024-09-14T07:00:43.086573+0000","last_fullsized":"2024-09-14T07:00:43.086573+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_clean_scrub_stamp":"2024-09-14T06:28:09.039132+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:00:29.799410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738768,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1732956,"kb_used_data":2760,"kb_used_omap":128,"kb_used_meta":1730047,"kb_avail":92638884,"statfs":{"total":96636764160,"available":94862217216,"internally_reserved":0,"allocated":2826240,"data_stored":1490488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131889,"internal_metadata":1771568335},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738769,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1786340,"kb_used_data":3344,"kb_used_omap":115,"kb_used_meta":1782860,"kb_avail":92585500,"statfs":{"total":96636764160,"available":94807552000,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118072,"internal_metadata":1825649352},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738770,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1267876,"kb_used_data":3344,"kb_used_omap":190,"kb_used_meta":1264321,"kb_avail":93103964,"statfs":{"total":96636764160,"available":95338459136,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":195559,"internal_metadata":1294664729},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:01:17.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T07:01:17.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T07:01:17.390 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-14T07:01:17.403 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1335,"stamp":"2024-09-14T07:01:16.456069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4787172,"kb_used_data":9448,"kb_used_omap":435,"kb_used_meta":4777228,"kb_avail":278328348,"statfs":{"total":289910292480,"available":285008228352,"internally_reserved":0,"allocated":9674752,"data_stored":5652238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":445520,"internal_metadata":4891882416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002343"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083819+0000","last_change":"2024-09-14T06:28:16.071845+0000","last_active":"2024-09-14T07:00:43.083819+0000","last_peered":"2024-09-14T07:00:43.083819+0000","last_clean":"2024-09-14T07:00:43.083819+0000","last_became_active":"2024-09-14T06:28:14.066720+0000","last_became_peered":"2024-09-14T06:28:14.066720+0000","last_unstale":"2024-09-14T07:00:43.083819+0000","last_undegraded":"2024-09-14T07:00:43.083819+0000","last_fullsized":"2024-09-14T07:00:43.083819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:18.865038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078953200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085739+0000","last_change":"2024-09-14T06:28:16.072384+0000","last_active":"2024-09-14T07:00:43.085739+0000","last_peered":"2024-09-14T07:00:43.085739+0000","last_clean":"2024-09-14T07:00:43.085739+0000","last_became_active":"2024-09-14T06:28:14.067568+0000","last_became_peered":"2024-09-14T06:28:14.067568+0000","last_unstale":"2024-09-14T07:00:43.085739+0000","last_undegraded":"2024-09-14T07:00:43.085739+0000","last_fullsized":"2024-09-14T07:00:43.085739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:25:25.337512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.087088+0000","last_change":"2024-09-14T06:28:16.072011+0000","last_active":"2024-09-14T07:00:43.087088+0000","last_peered":"2024-09-14T07:00:43.087088+0000","last_clean":"2024-09-14T07:00:43.087088+0000","last_became_active":"2024-09-14T06:28:14.065921+0000","last_became_peered":"2024-09-14T06:28:14.065921+0000","last_unstale":"2024-09-14T07:00:43.087088+0000","last_undegraded":"2024-09-14T07:00:43.087088+0000","last_fullsized":"2024-09-14T07:00:43.087088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:45:01.691891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092469000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085184+0000","last_change":"2024-09-14T06:28:16.072360+0000","last_active":"2024-09-14T07:00:43.085184+0000","last_peered":"2024-09-14T07:00:43.085184+0000","last_clean":"2024-09-14T07:00:43.085184+0000","last_became_active":"2024-09-14T06:28:14.067522+0000","last_became_peered":"2024-09-14T06:28:14.067522+0000","last_unstale":"2024-09-14T07:00:43.085184+0000","last_undegraded":"2024-09-14T07:00:43.085184+0000","last_fullsized":"2024-09-14T07:00:43.085184+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:56.670133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056764,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":693,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:42.752297+0000","last_change":"2024-09-14T06:28:16.069659+0000","last_active":"2024-09-14T07:00:42.752297+0000","last_peered":"2024-09-14T07:00:42.752297+0000","last_clean":"2024-09-14T07:00:42.752297+0000","last_became_active":"2024-09-14T06:28:14.065736+0000","last_became_peered":"2024-09-14T06:28:14.065736+0000","last_unstale":"2024-09-14T07:00:42.752297+0000","last_undegraded":"2024-09-14T07:00:42.752297+0000","last_fullsized":"2024-09-14T07:00:42.752297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:29.974020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010548370000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345473+0000","last_change":"2024-09-14T06:28:16.066424+0000","last_active":"2024-09-14T07:00:43.345473+0000","last_peered":"2024-09-14T07:00:43.345473+0000","last_clean":"2024-09-14T07:00:43.345473+0000","last_became_active":"2024-09-14T06:28:14.065677+0000","last_became_peered":"2024-09-14T06:28:14.065677+0000","last_unstale":"2024-09-14T07:00:43.345473+0000","last_undegraded":"2024-09-14T07:00:43.345473+0000","last_fullsized":"2024-09-14T07:00:43.345473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:36.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047781599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345573+0000","last_change":"2024-09-14T06:28:16.066327+0000","last_active":"2024-09-14T07:00:43.345573+0000","last_peered":"2024-09-14T07:00:43.345573+0000","last_clean":"2024-09-14T07:00:43.345573+0000","last_became_active":"2024-09-14T06:28:14.067147+0000","last_became_peered":"2024-09-14T06:28:14.067147+0000","last_unstale":"2024-09-14T07:00:43.345573+0000","last_undegraded":"2024-09-14T07:00:43.345573+0000","last_fullsized":"2024-09-14T07:00:43.345573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:19:13.309078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042511200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":692,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083496+0000","last_change":"2024-09-14T06:28:16.071663+0000","last_active":"2024-09-14T07:00:43.083496+0000","last_peered":"2024-09-14T07:00:43.083496+0000","last_clean":"2024-09-14T07:00:43.083496+0000","last_became_active":"2024-09-14T06:28:14.065753+0000","last_became_peered":"2024-09-14T06:28:14.065753+0000","last_unstale":"2024-09-14T07:00:43.083496+0000","last_undegraded":"2024-09-14T07:00:43.083496+0000","last_fullsized":"2024-09-14T07:00:43.083496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:38:56.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066619800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":760,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.086573+0000","last_change":"2024-09-14T06:28:10.140562+0000","last_active":"2024-09-14T07:00:43.086573+0000","last_peered":"2024-09-14T07:00:43.086573+0000","last_clean":"2024-09-14T07:00:43.086573+0000","last_became_active":"2024-09-14T06:28:10.139552+0000","last_became_peered":"2024-09-14T06:28:10.139552+0000","last_unstale":"2024-09-14T07:00:43.086573+0000","last_undegraded":"2024-09-14T07:00:43.086573+0000","last_fullsized":"2024-09-14T07:00:43.086573+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_clean_scrub_stamp":"2024-09-14T06:28:09.039132+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:00:29.799410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738768,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1732956,"kb_used_data":2760,"kb_used_omap":128,"kb_used_meta":1730047,"kb_avail":92638884,"statfs":{"total":96636764160,"available":94862217216,"internally_reserved":0,"allocated":2826240,"data_stored":1490488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131889,"internal_metadata":1771568335},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738769,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1786340,"kb_used_data":3344,"kb_used_omap":115,"kb_used_meta":1782860,"kb_avail":92585500,"statfs":{"total":96636764160,"available":94807552000,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118072,"internal_metadata":1825649352},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738770,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1267876,"kb_used_data":3344,"kb_used_omap":190,"kb_used_meta":1264321,"kb_avail":93103964,"statfs":{"total":96636764160,"available":95338459136,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":195559,"internal_metadata":1294664729},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:01:17.403 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-14T07:01:17.404 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T07:01:17.696 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T07:01:17.697 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-14T07:01:17.708 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1335,"stamp":"2024-09-14T07:01:16.456069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4787172,"kb_used_data":9448,"kb_used_omap":435,"kb_used_meta":4777228,"kb_avail":278328348,"statfs":{"total":289910292480,"available":285008228352,"internally_reserved":0,"allocated":9674752,"data_stored":5652238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":445520,"internal_metadata":4891882416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002343"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083819+0000","last_change":"2024-09-14T06:28:16.071845+0000","last_active":"2024-09-14T07:00:43.083819+0000","last_peered":"2024-09-14T07:00:43.083819+0000","last_clean":"2024-09-14T07:00:43.083819+0000","last_became_active":"2024-09-14T06:28:14.066720+0000","last_became_peered":"2024-09-14T06:28:14.066720+0000","last_unstale":"2024-09-14T07:00:43.083819+0000","last_undegraded":"2024-09-14T07:00:43.083819+0000","last_fullsized":"2024-09-14T07:00:43.083819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:18.865038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078953200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085739+0000","last_change":"2024-09-14T06:28:16.072384+0000","last_active":"2024-09-14T07:00:43.085739+0000","last_peered":"2024-09-14T07:00:43.085739+0000","last_clean":"2024-09-14T07:00:43.085739+0000","last_became_active":"2024-09-14T06:28:14.067568+0000","last_became_peered":"2024-09-14T06:28:14.067568+0000","last_unstale":"2024-09-14T07:00:43.085739+0000","last_undegraded":"2024-09-14T07:00:43.085739+0000","last_fullsized":"2024-09-14T07:00:43.085739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:25:25.337512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.087088+0000","last_change":"2024-09-14T06:28:16.072011+0000","last_active":"2024-09-14T07:00:43.087088+0000","last_peered":"2024-09-14T07:00:43.087088+0000","last_clean":"2024-09-14T07:00:43.087088+0000","last_became_active":"2024-09-14T06:28:14.065921+0000","last_became_peered":"2024-09-14T06:28:14.065921+0000","last_unstale":"2024-09-14T07:00:43.087088+0000","last_undegraded":"2024-09-14T07:00:43.087088+0000","last_fullsized":"2024-09-14T07:00:43.087088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:45:01.691891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092469000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085184+0000","last_change":"2024-09-14T06:28:16.072360+0000","last_active":"2024-09-14T07:00:43.085184+0000","last_peered":"2024-09-14T07:00:43.085184+0000","last_clean":"2024-09-14T07:00:43.085184+0000","last_became_active":"2024-09-14T06:28:14.067522+0000","last_became_peered":"2024-09-14T06:28:14.067522+0000","last_unstale":"2024-09-14T07:00:43.085184+0000","last_undegraded":"2024-09-14T07:00:43.085184+0000","last_fullsized":"2024-09-14T07:00:43.085184+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:56.670133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056764,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":693,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:42.752297+0000","last_change":"2024-09-14T06:28:16.069659+0000","last_active":"2024-09-14T07:00:42.752297+0000","last_peered":"2024-09-14T07:00:42.752297+0000","last_clean":"2024-09-14T07:00:42.752297+0000","last_became_active":"2024-09-14T06:28:14.065736+0000","last_became_peered":"2024-09-14T06:28:14.065736+0000","last_unstale":"2024-09-14T07:00:42.752297+0000","last_undegraded":"2024-09-14T07:00:42.752297+0000","last_fullsized":"2024-09-14T07:00:42.752297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:29.974020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010548370000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345473+0000","last_change":"2024-09-14T06:28:16.066424+0000","last_active":"2024-09-14T07:00:43.345473+0000","last_peered":"2024-09-14T07:00:43.345473+0000","last_clean":"2024-09-14T07:00:43.345473+0000","last_became_active":"2024-09-14T06:28:14.065677+0000","last_became_peered":"2024-09-14T06:28:14.065677+0000","last_unstale":"2024-09-14T07:00:43.345473+0000","last_undegraded":"2024-09-14T07:00:43.345473+0000","last_fullsized":"2024-09-14T07:00:43.345473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:36.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047781599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345573+0000","last_change":"2024-09-14T06:28:16.066327+0000","last_active":"2024-09-14T07:00:43.345573+0000","last_peered":"2024-09-14T07:00:43.345573+0000","last_clean":"2024-09-14T07:00:43.345573+0000","last_became_active":"2024-09-14T06:28:14.067147+0000","last_became_peered":"2024-09-14T06:28:14.067147+0000","last_unstale":"2024-09-14T07:00:43.345573+0000","last_undegraded":"2024-09-14T07:00:43.345573+0000","last_fullsized":"2024-09-14T07:00:43.345573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:19:13.309078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042511200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":692,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083496+0000","last_change":"2024-09-14T06:28:16.071663+0000","last_active":"2024-09-14T07:00:43.083496+0000","last_peered":"2024-09-14T07:00:43.083496+0000","last_clean":"2024-09-14T07:00:43.083496+0000","last_became_active":"2024-09-14T06:28:14.065753+0000","last_became_peered":"2024-09-14T06:28:14.065753+0000","last_unstale":"2024-09-14T07:00:43.083496+0000","last_undegraded":"2024-09-14T07:00:43.083496+0000","last_fullsized":"2024-09-14T07:00:43.083496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:38:56.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066619800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":760,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.086573+0000","last_change":"2024-09-14T06:28:10.140562+0000","last_active":"2024-09-14T07:00:43.086573+0000","last_peered":"2024-09-14T07:00:43.086573+0000","last_clean":"2024-09-14T07:00:43.086573+0000","last_became_active":"2024-09-14T06:28:10.139552+0000","last_became_peered":"2024-09-14T06:28:10.139552+0000","last_unstale":"2024-09-14T07:00:43.086573+0000","last_undegraded":"2024-09-14T07:00:43.086573+0000","last_fullsized":"2024-09-14T07:00:43.086573+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_clean_scrub_stamp":"2024-09-14T06:28:09.039132+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:00:29.799410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738768,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1732956,"kb_used_data":2760,"kb_used_omap":128,"kb_used_meta":1730047,"kb_avail":92638884,"statfs":{"total":96636764160,"available":94862217216,"internally_reserved":0,"allocated":2826240,"data_stored":1490488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131889,"internal_metadata":1771568335},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738769,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1786340,"kb_used_data":3344,"kb_used_omap":115,"kb_used_meta":1782860,"kb_avail":92585500,"statfs":{"total":96636764160,"available":94807552000,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118072,"internal_metadata":1825649352},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738770,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1267876,"kb_used_data":3344,"kb_used_omap":190,"kb_used_meta":1264321,"kb_avail":93103964,"statfs":{"total":96636764160,"available":95338459136,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":195559,"internal_metadata":1294664729},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:01:17.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T07:01:18.008 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T07:01:18.008 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":343,"fsid":"5c072185-95f2-48ae-9e2f-90187671cc6f","created":"2024-09-14T06:27:57.438728+0000","modified":"2024-09-14T07:00:42.744576+0000","last_up_change":"2024-09-14T06:28:07.395120+0000","last_in_change":"2024-09-14T06:27:58.309768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T06:28:08.189460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T06:28:12.957687+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"71e380e7-23ee-4d9f-a5dc-914c6a7fb873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6809","nonce":1600313121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6811","nonce":1600313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6815","nonce":1600313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":1600313121},{"type":"v1","addr":"172.21.15.1:6813","nonce":1600313121}]},"public_addr":"172.21.15.1:6809/1600313121","cluster_addr":"172.21.15.1:6811/1600313121","heartbeat_back_addr":"172.21.15.1:6815/1600313121","heartbeat_front_addr":"172.21.15.1:6813/1600313121","state":["exists","up"]},{"osd":1,"uuid":"d8033edb-e276-4d40-a9a4-32ec43ac783b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6801","nonce":1512690988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6803","nonce":1512690988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6807","nonce":1512690988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":1512690988},{"type":"v1","addr":"172.21.15.1:6805","nonce":1512690988}]},"public_addr":"172.21.15.1:6801/1512690988","cluster_addr":"172.21.15.1:6803/1512690988","heartbeat_back_addr":"172.21.15.1:6807/1512690988","heartbeat_front_addr":"172.21.15.1:6805/1512690988","state":["exists","up"]},{"osd":2,"uuid":"709cab32-6621-4a1e-8a40-e2a732d2b50c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6817","nonce":2637205829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6819","nonce":2637205829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6823","nonce":2637205829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":2637205829},{"type":"v1","addr":"172.21.15.1:6821","nonce":2637205829}]},"public_addr":"172.21.15.1:6817/2637205829","cluster_addr":"172.21.15.1:6819/2637205829","heartbeat_back_addr":"172.21.15.1:6823/2637205829","heartbeat_front_addr":"172.21.15.1:6821/2637205829","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.146261+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.369465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T06:28:04.402451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":133,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T07:01:19.022 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-14T07:01:19.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-14T07:01:19.162 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-14T07:01:19.162 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T07:01:19.162 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-14T07:01:19.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-14T07:01:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-09-14T07:01:19.484 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-14T07:01:19.484 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-14T07:01:19.611 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-14T07:01:19.611 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T07:01:19.611 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-14T07:01:19.622 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-14T07:01:19.919 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-09-14T07:01:19.932 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-14T07:01:19.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-14T07:01:20.058 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-14T07:01:20.058 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T07:01:20.058 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-14T07:01:20.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-14T07:01:20.366 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-09-14T07:01:20.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T07:01:20.672 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T07:01:20.673 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-14T07:01:20.685 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1337,"stamp":"2024-09-14T07:01:20.456732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4787172,"kb_used_data":9448,"kb_used_omap":435,"kb_used_meta":4777228,"kb_avail":278328348,"statfs":{"total":289910292480,"available":285008228352,"internally_reserved":0,"allocated":9674752,"data_stored":5652238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":445520,"internal_metadata":4891882416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002222"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083819+0000","last_change":"2024-09-14T06:28:16.071845+0000","last_active":"2024-09-14T07:00:43.083819+0000","last_peered":"2024-09-14T07:00:43.083819+0000","last_clean":"2024-09-14T07:00:43.083819+0000","last_became_active":"2024-09-14T06:28:14.066720+0000","last_became_peered":"2024-09-14T06:28:14.066720+0000","last_unstale":"2024-09-14T07:00:43.083819+0000","last_undegraded":"2024-09-14T07:00:43.083819+0000","last_fullsized":"2024-09-14T07:00:43.083819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:18.865038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078953200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085739+0000","last_change":"2024-09-14T06:28:16.072384+0000","last_active":"2024-09-14T07:00:43.085739+0000","last_peered":"2024-09-14T07:00:43.085739+0000","last_clean":"2024-09-14T07:00:43.085739+0000","last_became_active":"2024-09-14T06:28:14.067568+0000","last_became_peered":"2024-09-14T06:28:14.067568+0000","last_unstale":"2024-09-14T07:00:43.085739+0000","last_undegraded":"2024-09-14T07:00:43.085739+0000","last_fullsized":"2024-09-14T07:00:43.085739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:25:25.337512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.087088+0000","last_change":"2024-09-14T06:28:16.072011+0000","last_active":"2024-09-14T07:00:43.087088+0000","last_peered":"2024-09-14T07:00:43.087088+0000","last_clean":"2024-09-14T07:00:43.087088+0000","last_became_active":"2024-09-14T06:28:14.065921+0000","last_became_peered":"2024-09-14T06:28:14.065921+0000","last_unstale":"2024-09-14T07:00:43.087088+0000","last_undegraded":"2024-09-14T07:00:43.087088+0000","last_fullsized":"2024-09-14T07:00:43.087088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:45:01.691891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092469000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":691,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.085184+0000","last_change":"2024-09-14T06:28:16.072360+0000","last_active":"2024-09-14T07:00:43.085184+0000","last_peered":"2024-09-14T07:00:43.085184+0000","last_clean":"2024-09-14T07:00:43.085184+0000","last_became_active":"2024-09-14T06:28:14.067522+0000","last_became_peered":"2024-09-14T06:28:14.067522+0000","last_unstale":"2024-09-14T07:00:43.085184+0000","last_undegraded":"2024-09-14T07:00:43.085184+0000","last_fullsized":"2024-09-14T07:00:43.085184+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:56.670133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056764,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":693,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:42.752297+0000","last_change":"2024-09-14T06:28:16.069659+0000","last_active":"2024-09-14T07:00:42.752297+0000","last_peered":"2024-09-14T07:00:42.752297+0000","last_clean":"2024-09-14T07:00:42.752297+0000","last_became_active":"2024-09-14T06:28:14.065736+0000","last_became_peered":"2024-09-14T06:28:14.065736+0000","last_unstale":"2024-09-14T07:00:42.752297+0000","last_undegraded":"2024-09-14T07:00:42.752297+0000","last_fullsized":"2024-09-14T07:00:42.752297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:29.974020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010548370000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345473+0000","last_change":"2024-09-14T06:28:16.066424+0000","last_active":"2024-09-14T07:00:43.345473+0000","last_peered":"2024-09-14T07:00:43.345473+0000","last_clean":"2024-09-14T07:00:43.345473+0000","last_became_active":"2024-09-14T06:28:14.065677+0000","last_became_peered":"2024-09-14T06:28:14.065677+0000","last_unstale":"2024-09-14T07:00:43.345473+0000","last_undegraded":"2024-09-14T07:00:43.345473+0000","last_fullsized":"2024-09-14T07:00:43.345473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:36.270676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047781599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":690,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.345573+0000","last_change":"2024-09-14T06:28:16.066327+0000","last_active":"2024-09-14T07:00:43.345573+0000","last_peered":"2024-09-14T07:00:43.345573+0000","last_clean":"2024-09-14T07:00:43.345573+0000","last_became_active":"2024-09-14T06:28:14.067147+0000","last_became_peered":"2024-09-14T06:28:14.067147+0000","last_unstale":"2024-09-14T07:00:43.345573+0000","last_undegraded":"2024-09-14T07:00:43.345573+0000","last_fullsized":"2024-09-14T07:00:43.345573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:19:13.309078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042511200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":692,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.083496+0000","last_change":"2024-09-14T06:28:16.071663+0000","last_active":"2024-09-14T07:00:43.083496+0000","last_peered":"2024-09-14T07:00:43.083496+0000","last_clean":"2024-09-14T07:00:43.083496+0000","last_became_active":"2024-09-14T06:28:14.065753+0000","last_became_peered":"2024-09-14T06:28:14.065753+0000","last_unstale":"2024-09-14T07:00:43.083496+0000","last_undegraded":"2024-09-14T07:00:43.083496+0000","last_fullsized":"2024-09-14T07:00:43.083496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:13.052453+0000","last_clean_scrub_stamp":"2024-09-14T06:28:13.052453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:38:56.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066619800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":760,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:00:43.086573+0000","last_change":"2024-09-14T06:28:10.140562+0000","last_active":"2024-09-14T07:00:43.086573+0000","last_peered":"2024-09-14T07:00:43.086573+0000","last_clean":"2024-09-14T07:00:43.086573+0000","last_became_active":"2024-09-14T06:28:10.139552+0000","last_became_peered":"2024-09-14T06:28:10.139552+0000","last_unstale":"2024-09-14T07:00:43.086573+0000","last_undegraded":"2024-09-14T07:00:43.086573+0000","last_fullsized":"2024-09-14T07:00:43.086573+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:28:09.039132+0000","last_clean_scrub_stamp":"2024-09-14T06:28:09.039132+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:00:29.799410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738769,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1732956,"kb_used_data":2760,"kb_used_omap":128,"kb_used_meta":1730047,"kb_avail":92638884,"statfs":{"total":96636764160,"available":94862217216,"internally_reserved":0,"allocated":2826240,"data_stored":1490488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131889,"internal_metadata":1771568335},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738770,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1786340,"kb_used_data":3344,"kb_used_omap":115,"kb_used_meta":1782860,"kb_avail":92585500,"statfs":{"total":96636764160,"available":94807552000,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118072,"internal_metadata":1825649352},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738771,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1267876,"kb_used_data":3344,"kb_used_omap":190,"kb_used_meta":1264321,"kb_avail":93103964,"statfs":{"total":96636764160,"available":95338459136,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":195559,"internal_metadata":1294664729},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:01:20.686 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.686 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.686 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.686 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.686 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.686 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.686 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.687 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-14T06:28:13.052453+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=13, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.687 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-14T06:28:09.039132+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=28, tm_sec=9, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=7, tm_min=1, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T07:01:20.687 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-14T07:01:40.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T07:01:41.000 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-14T07:01:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-14T07:01:41.013 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1347,"stamp":"2024-09-14T07:01:40.460665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4787160,"kb_used_data":9448,"kb_used_omap":435,"kb_used_meta":4777228,"kb_avail":278328360,"statfs":{"total":289910292480,"available":285008240640,"internally_reserved":0,"allocated":9674752,"data_stored":5652238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":445520,"internal_metadata":4891882416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002456"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":699,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:25.348174+0000","last_change":"2024-09-14T07:01:25.348174+0000","last_active":"2024-09-14T07:01:25.348174+0000","last_peered":"2024-09-14T07:01:25.348174+0000","last_clean":"2024-09-14T07:01:25.348174+0000","last_became_active":"2024-09-14T06:28:14.066720+0000","last_became_peered":"2024-09-14T06:28:14.066720+0000","last_unstale":"2024-09-14T07:01:25.348174+0000","last_undegraded":"2024-09-14T07:01:25.348174+0000","last_fullsized":"2024-09-14T07:01:25.348174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:01:25.348075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:01:25.348075+0000","last_clean_scrub_stamp":"2024-09-14T07:01:25.348075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:18.853654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078953200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":699,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:23.407708+0000","last_change":"2024-09-14T07:01:23.407708+0000","last_active":"2024-09-14T07:01:23.407708+0000","last_peered":"2024-09-14T07:01:23.407708+0000","last_clean":"2024-09-14T07:01:23.407708+0000","last_became_active":"2024-09-14T06:28:14.067568+0000","last_became_peered":"2024-09-14T06:28:14.067568+0000","last_unstale":"2024-09-14T07:01:23.407708+0000","last_undegraded":"2024-09-14T07:01:23.407708+0000","last_fullsized":"2024-09-14T07:01:23.407708+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:01:23.407612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:01:23.407612+0000","last_clean_scrub_stamp":"2024-09-14T07:01:23.407612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:04:37.726346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":699,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:22.370479+0000","last_change":"2024-09-14T07:01:22.370479+0000","last_active":"2024-09-14T07:01:22.370479+0000","last_peered":"2024-09-14T07:01:22.370479+0000","last_clean":"2024-09-14T07:01:22.370479+0000","last_became_active":"2024-09-14T06:28:14.065921+0000","last_became_peered":"2024-09-14T06:28:14.065921+0000","last_unstale":"2024-09-14T07:01:22.370479+0000","last_undegraded":"2024-09-14T07:01:22.370479+0000","last_fullsized":"2024-09-14T07:01:22.370479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:01:22.370373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:01:22.370373+0000","last_clean_scrub_stamp":"2024-09-14T07:01:22.370373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T17:02:00.568600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092469000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":700,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:20.351332+0000","last_change":"2024-09-14T07:01:20.351290+0000","last_active":"2024-09-14T07:01:20.351332+0000","last_peered":"2024-09-14T07:01:20.351332+0000","last_clean":"2024-09-14T07:01:20.351332+0000","last_became_active":"2024-09-14T06:28:14.067522+0000","last_became_peered":"2024-09-14T06:28:14.067522+0000","last_unstale":"2024-09-14T07:01:20.351332+0000","last_undegraded":"2024-09-14T07:01:20.351332+0000","last_fullsized":"2024-09-14T07:01:20.351332+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:01:20.351234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:01:20.351234+0000","last_clean_scrub_stamp":"2024-09-14T07:01:20.351234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T18:21:49.642748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056764,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"343'3","reported_seq":703,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:19.556212+0000","last_change":"2024-09-14T07:01:19.556149+0000","last_active":"2024-09-14T07:01:19.556212+0000","last_peered":"2024-09-14T07:01:19.556212+0000","last_clean":"2024-09-14T07:01:19.556212+0000","last_became_active":"2024-09-14T06:28:14.065736+0000","last_became_peered":"2024-09-14T06:28:14.065736+0000","last_unstale":"2024-09-14T07:01:19.556212+0000","last_undegraded":"2024-09-14T07:01:19.556212+0000","last_fullsized":"2024-09-14T07:01:19.556212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'3","last_scrub_stamp":"2024-09-14T07:01:19.556082+0000","last_deep_scrub":"343'3","last_deep_scrub_stamp":"2024-09-14T07:01:19.556082+0000","last_clean_scrub_stamp":"2024-09-14T07:01:19.556082+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T17:27:18.197722+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010548370000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":698,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:22.323983+0000","last_change":"2024-09-14T07:01:22.323930+0000","last_active":"2024-09-14T07:01:22.323983+0000","last_peered":"2024-09-14T07:01:22.323983+0000","last_clean":"2024-09-14T07:01:22.323983+0000","last_became_active":"2024-09-14T06:28:14.065677+0000","last_became_peered":"2024-09-14T06:28:14.065677+0000","last_unstale":"2024-09-14T07:01:22.323983+0000","last_undegraded":"2024-09-14T07:01:22.323983+0000","last_fullsized":"2024-09-14T07:01:22.323983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:01:22.323833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:01:22.323833+0000","last_clean_scrub_stamp":"2024-09-14T07:01:22.323833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:23:16.026731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047781599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":698,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:21.339068+0000","last_change":"2024-09-14T07:01:21.339068+0000","last_active":"2024-09-14T07:01:21.339068+0000","last_peered":"2024-09-14T07:01:21.339068+0000","last_clean":"2024-09-14T07:01:21.339068+0000","last_became_active":"2024-09-14T06:28:14.067147+0000","last_became_peered":"2024-09-14T06:28:14.067147+0000","last_unstale":"2024-09-14T07:01:21.339068+0000","last_undegraded":"2024-09-14T07:01:21.339068+0000","last_fullsized":"2024-09-14T07:01:21.339068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T07:01:21.338973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T07:01:21.338973+0000","last_clean_scrub_stamp":"2024-09-14T07:01:21.338973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:21:18.632380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00042511200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"343'2","reported_seq":702,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:21.338447+0000","last_change":"2024-09-14T07:01:21.338384+0000","last_active":"2024-09-14T07:01:21.338447+0000","last_peered":"2024-09-14T07:01:21.338447+0000","last_clean":"2024-09-14T07:01:21.338447+0000","last_became_active":"2024-09-14T06:28:14.065753+0000","last_became_peered":"2024-09-14T06:28:14.065753+0000","last_unstale":"2024-09-14T07:01:21.338447+0000","last_undegraded":"2024-09-14T07:01:21.338447+0000","last_fullsized":"2024-09-14T07:01:21.338447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'2","last_scrub_stamp":"2024-09-14T07:01:21.338252+0000","last_deep_scrub":"343'2","last_deep_scrub_stamp":"2024-09-14T07:01:21.338252+0000","last_clean_scrub_stamp":"2024-09-14T07:01:21.338252+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:10:35.621230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00066619800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"343'54","reported_seq":770,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-14T07:01:24.383784+0000","last_change":"2024-09-14T07:01:24.383784+0000","last_active":"2024-09-14T07:01:24.383784+0000","last_peered":"2024-09-14T07:01:24.383784+0000","last_clean":"2024-09-14T07:01:24.383784+0000","last_became_active":"2024-09-14T06:28:10.139552+0000","last_became_peered":"2024-09-14T06:28:10.139552+0000","last_unstale":"2024-09-14T07:01:24.383784+0000","last_undegraded":"2024-09-14T07:01:24.383784+0000","last_fullsized":"2024-09-14T07:01:24.383784+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'54","last_scrub_stamp":"2024-09-14T07:01:24.383680+0000","last_deep_scrub":"343'54","last_deep_scrub_stamp":"2024-09-14T07:01:24.383680+0000","last_clean_scrub_stamp":"2024-09-14T07:01:24.383680+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:49:45.687156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738773,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1732952,"kb_used_data":2760,"kb_used_omap":128,"kb_used_meta":1730047,"kb_avail":92638888,"statfs":{"total":96636764160,"available":94862221312,"internally_reserved":0,"allocated":2826240,"data_stored":1490488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131889,"internal_metadata":1771568335},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738774,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1786336,"kb_used_data":3344,"kb_used_omap":115,"kb_used_meta":1782860,"kb_avail":92585504,"statfs":{"total":96636764160,"available":94807556096,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":118072,"internal_metadata":1825649352},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738775,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1267872,"kb_used_data":3344,"kb_used_omap":190,"kb_used_meta":1264321,"kb_avail":93103968,"statfs":{"total":96636764160,"available":95338463232,"internally_reserved":0,"allocated":3424256,"data_stored":2080875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":195559,"internal_metadata":1294664729},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T07:01:41.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-14T07:01:41.328 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-14T07:01:41.329 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-14T07:01:41.329 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-14T07:01:41.329 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:41.420 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-14T07:01:41.421 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-14T07:01:41.421 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:41.497 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-14T07:01:41.497 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-14T07:01:41.497 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:41.567 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-14T07:01:41.567 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-14T07:01:41.567 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-14T07:01:41.567 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:41.594 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-14T07:01:41.594 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-14T07:01:41.594 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-14T07:01:41.594 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T07:01:41.647 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-14T07:01:41.647 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-14T07:01:41.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T07:01:41.751 INFO:teuthology.orchestra.run.smithi001.stdout:2024-09-14T06:31:04.474787+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-14T07:01:41.752 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-14T07:01:41.752 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T07:01:41.779 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T07:01:41.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[WRN\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T07:01:41.914 INFO:teuthology.orchestra.run.smithi001.stdout:2024-09-14T06:31:04.474787+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-14T07:01:41.915 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T07:01:41.915 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-14T07:01:42.364 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T07:01:42.364 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-14T07:01:42.608 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T07:01:42.608 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-14T07:01:42.876 INFO:tasks.ceph:Archiving mon data... 2024-09-14T07:01:42.877 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/data/cluster1.mon.a.tgz 2024-09-14T07:01:42.877 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-14T07:01:42.928 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.AHN8x9yfJ1 2024-09-14T07:01:42.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.AHN8x9yfJ1 2024-09-14T07:01:43.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.AHN8x9yfJ1 2024-09-14T07:01:43.181 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.AHN8x9yfJ1 is 422KB 2024-09-14T07:01:43.252 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.AHN8x9yfJ1 2024-09-14T07:01:43.268 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-14T07:01:43.269 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T07:01:43.310 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T07:01:43.398 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T07:01:43.434 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T07:01:43.434 INFO:tasks.ceph:Archiving crash dumps... 2024-09-14T07:01:43.436 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/remote/smithi001/crash 2024-09-14T07:01:43.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T07:01:43.464 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/remote/smithi170/crash 2024-09-14T07:01:43.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T07:01:43.502 INFO:tasks.ceph:Compressing logs... 2024-09-14T07:01:43.503 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T07:01:43.509 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35793.log 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37751.log 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/cluster1-client.admin.35793.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36233.log 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35793.log.gz 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37751.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.admin.37751.log.gz /var/log/ceph/cluster1-mon.a.log 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36126.log 2024-09-14T07:01:43.555 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36233.log.gz 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27619.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35514.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36126.log.gz 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.27619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37583.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27619.log.gz 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35514.log.gz 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35999.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37583.log.gz 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37541.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37625.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35999.log.gz 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32296.log 2024-09-14T07:01:43.556 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37541.log.gz 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37457.log 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37625.log.gz 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32098.log 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32296.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32296.log.gz 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36423.log 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37457.log.gz 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35742.log 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32098.log.gz 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37793.log 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36423.log.gz 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-14T07:01:43.557 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37793.log.gz 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37415.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37833.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37415.log.gz 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36381.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37833.log.gz 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37709.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35919.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36381.log.gz 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36087.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37709.log.gz 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36213.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35919.log.gz 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32435.log 2024-09-14T07:01:43.558 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36087.log.gz 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36213.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36213.log.gz 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37667.log 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32435.log.gz 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32256.log 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35877.log 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37667.log.gz 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35835.log 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32256.log.gz 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37373.log 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35877.log.gz 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35835.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.35835.log.gz 2024-09-14T07:01:43.559 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36127.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37373.log.gz 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32336.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37499.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36127.log.gz 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36001.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32336.log.gz 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36000.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37499.log.gz 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35717.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36001.log.gz 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36000.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-14T07:01:43.560 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36000.log.gz 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36255.log 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35717.log.gz 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32507.log 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35661.log 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36255.log.gz 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36339.log 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35661.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35661.log.gz 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37331.log 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36339.log.gz 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37331.log.gz 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr: 25.2% -- replaced with /var/log/ceph/cluster1-client.admin.32507.log.gz 2024-09-14T07:01:43.561 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-14T07:01:43.592 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83761.log 2024-09-14T07:01:43.592 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40236.log 2024-09-14T07:01:43.593 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42644.log 2024-09-14T07:01:43.593 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82265.log 2024-09-14T07:01:43.593 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83761.log.gz 2024-09-14T07:01:43.594 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40236.log: /var/log/ceph/cluster2-client.mirror.42644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82913.log 2024-09-14T07:01:43.594 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67313.log 2024-09-14T07:01:43.620 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86027.log 2024-09-14T07:01:43.620 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82265.log.gz 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.67313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64566.log 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46064.log 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86027.log: /var/log/ceph/cluster2-client.mirror.64566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89461.log 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91413.log 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67313.log.gz 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82913.log.gz 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65807.log 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89461.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89461.log.gz 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84924.log 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65807.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86027.log.gz 2024-09-14T07:01:43.621 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.91413.log.gz 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82287.log 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65807.log.gz 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89187.log 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41520.log 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89187.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82287.log.gz 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87314.log 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89187.log.gz 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69208.log 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84924.log.gz 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86707.log 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87314.log.gz 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38072.log 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41520.log.gz 2024-09-14T07:01:43.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86707.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69208.log.gz 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77262.log 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86707.log.gz 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45708.log 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38072.log.gz 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64750.log 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89608.log 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.64750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87900.log 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89608.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64750.log.gz 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41318.log 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89608.log.gz 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91145.log 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87900.log: /var/log/ceph/cluster1-client.mirror.41318.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87900.log.gz 2024-09-14T07:01:43.623 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73527.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43991.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr: 94.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77262.log.gz 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.73527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62345.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40702.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43991.log: /var/log/ceph/cluster1-client.mirror.62345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41591.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91145.log.gz 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.40702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86819.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr: 91.8%/var/log/ceph/cluster1-client.mirror.41591.log: -- replaced with /var/log/ceph/cluster1-client.mirror.73527.log.gz 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82194.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44886.log 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86819.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40236.log.gz 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62345.log.gz 2024-09-14T07:01:43.624 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82194.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86819.log.gz 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78505.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.44886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88125.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82194.log.gz 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88530.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.40702.log.gz 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41591.log.gz 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73507.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88125.log.gz 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65007.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88530.log.gz 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87067.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53710.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65007.log: /var/log/ceph/cluster1-client.mirror.87067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76785.log 2024-09-14T07:01:43.625 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87067.log.gz 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.53710.log: gzip 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73507.log.gz 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41701.log 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62268.log 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76785.log.gz 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41701.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65007.log.gz 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74307.log 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85612.log 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62268.log.gz 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.74307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71475.log 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85612.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74307.log.gz 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91166.log 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85612.log.gz 2024-09-14T07:01:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.71475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73141.log 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91166.log: gzip 83.3% -5 --verbose 88.0% -- /var/log/ceph/cluster2-client.mirror.62646.log -- replaced with /var/log/ceph/cluster1-client.mirror.71475.log.gz 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.91166.log.gz 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.73141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86977.log 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73163.log 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86977.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62646.log.gz 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36449.log 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86977.log.gz 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73163.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46952.log 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36449.log: gzip -5 91.9% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.73141.log.gz /var/log/ceph/cluster2-client.mirror.83428.log 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42644.log.gz 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36449.log.gz 2024-09-14T07:01:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66282.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52932.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83428.log: /var/log/ceph/cluster2-client.mirror.66282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75833.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66282.log.gz 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.52932.log: gzip -5 84.4% --verbose -- /var/log/ceph/cluster2-client.mirror.74020.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52932.log.gz 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58809.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.74020.log: --verbose 90.6% -- /var/log/ceph/cluster2-client.mirror.79134.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.83428.log.gz 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.58809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37667.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79134.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79134.log.gz 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75833.log.gz 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91924.log 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37667.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37667.log.gz 2024-09-14T07:01:43.628 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41791.log 2024-09-14T07:01:43.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91924.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.91924.log.gz 2024-09-14T07:01:43.629 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72265.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.41791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41226.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.72265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82751.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.41226.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72265.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81440.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82751.log: gzip/var/log/ceph/cluster1-client.mirror.81440.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82751.log.gz 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41616.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.81440.log.gz 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85561.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89269.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85561.log: /var/log/ceph/cluster2-client.mirror.41616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67335.log 2024-09-14T07:01:43.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89269.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41226.log.gz 2024-09-14T07:01:43.635 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89269.log.gz 2024-09-14T07:01:43.635 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85561.log.gz 2024-09-14T07:01:43.635 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89859.log 2024-09-14T07:01:43.638 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67335.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41318.log.gz 2024-09-14T07:01:43.638 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82871.log 2024-09-14T07:01:43.638 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89859.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89859.log.gz 2024-09-14T07:01:43.638 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71500.log 2024-09-14T07:01:43.638 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82871.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.82871.log.gz 2024-09-14T07:01:43.638 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71928.log 2024-09-14T07:01:43.640 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.71500.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71500.log.gz 2024-09-14T07:01:43.647 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46107.log 2024-09-14T07:01:43.662 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.71928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72399.log 2024-09-14T07:01:43.665 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46107.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.41791.log.gz 2024-09-14T07:01:43.670 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69243.log 2024-09-14T07:01:43.675 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.72399.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45708.log.gz 2024-09-14T07:01:43.675 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46107.log.gz 2024-09-14T07:01:43.676 INFO:teuthology.orchestra.run.smithi001.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.35742.log.gz 2024-09-14T07:01:43.678 INFO:teuthology.orchestra.run.smithi170.stderr: 96.8% 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43991.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.41701.log.gz 2024-09-14T07:01:43.678 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.681 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81714.log 2024-09-14T07:01:43.682 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69243.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69243.log.gz 2024-09-14T07:01:43.685 INFO:teuthology.orchestra.run.smithi001.stderr: 94.1% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-14T07:01:43.702 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40387.log 2024-09-14T07:01:43.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81714.log: 87.9% 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46064.log.gz 2024-09-14T07:01:43.715 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32418.log 2024-09-14T07:01:43.715 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40387.log: -- replaced with /var/log/ceph/cluster2-client.mirror.81714.log.gz 2024-09-14T07:01:43.716 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40387.log.gz 2024-09-14T07:01:43.719 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44886.log.gz 2024-09-14T07:01:43.733 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63760.log 2024-09-14T07:01:43.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.32418.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32418.log.gz 2024-09-14T07:01:43.735 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41616.log.gz 2024-09-14T07:01:43.745 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46596.log 2024-09-14T07:01:43.752 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63760.log: 92.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38516.log 2024-09-14T07:01:43.757 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.46596.log: -- replaced with /var/log/ceph/cluster2-client.mirror.63760.log.gz 2024-09-14T07:01:43.774 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87675.log 2024-09-14T07:01:43.774 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.1.38516.log: -- replaced with /var/log/ceph/cluster2-client.mirror.67335.log.gz 2024-09-14T07:01:43.774 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46596.log.gz 2024-09-14T07:01:43.775 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82216.log 2024-09-14T07:01:43.775 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87675.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87675.log.gz 2024-09-14T07:01:43.775 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86864.log 2024-09-14T07:01:43.776 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82216.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82216.log.gz 2024-09-14T07:01:43.776 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66003.log 2024-09-14T07:01:43.776 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86864.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86864.log.gz 2024-09-14T07:01:43.780 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38312.log 2024-09-14T07:01:43.784 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66003.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73163.log.gz 2024-09-14T07:01:43.788 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78505.log.gz 2024-09-14T07:01:43.789 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77606.log 2024-09-14T07:01:43.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41999.log 2024-09-14T07:01:43.791 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77606.log: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66003.log.gz 2024-09-14T07:01:43.791 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77606.log.gz 2024-09-14T07:01:43.793 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.41999.log: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38312.log.gz 2024-09-14T07:01:43.793 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.57678.log 2024-09-14T07:01:43.795 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41555.log 2024-09-14T07:01:43.795 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.57678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43264.log 2024-09-14T07:01:43.795 INFO:teuthology.orchestra.run.smithi170.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57678.log.gz 2024-09-14T07:01:43.795 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78077.log 2024-09-14T07:01:43.800 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88283.log 2024-09-14T07:01:43.800 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41555.log.gz 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78077.log: /var/log/ceph/cluster2-client.mirror.43264.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43264.log.gz 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47599.log 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88283.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88283.log.gz 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45126.log 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47599.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78077.log.gz 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91771.log 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81462.log 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38157.log 2024-09-14T07:01:43.801 INFO:teuthology.orchestra.run.smithi170.stderr: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91771.log.gz 2024-09-14T07:01:43.807 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81462.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64566.log.gz 2024-09-14T07:01:43.808 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37973.log 2024-09-14T07:01:43.808 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38157.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57703.log 2024-09-14T07:01:43.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37973.log: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38157.log.gz 2024-09-14T07:01:43.810 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83881.log 2024-09-14T07:01:43.810 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0%/var/log/ceph/cluster2-client.mirror.57703.log: -- replaced with /var/log/ceph/cluster1-client.admin.37973.log.gz 2024-09-14T07:01:43.810 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87449.log 2024-09-14T07:01:43.811 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83881.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83881.log.gz 2024-09-14T07:01:43.811 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87653.log 2024-09-14T07:01:43.814 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87449.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87449.log.gz 2024-09-14T07:01:43.814 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90006.log 2024-09-14T07:01:43.814 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87653.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87653.log.gz 2024-09-14T07:01:43.814 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90907.log 2024-09-14T07:01:43.815 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90006.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90006.log.gz 2024-09-14T07:01:43.815 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42917.log 2024-09-14T07:01:43.819 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90907.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90907.log.gz 2024-09-14T07:01:43.819 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74339.log 2024-09-14T07:01:43.820 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68536.log 2024-09-14T07:01:43.821 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.74339.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74339.log.gz 2024-09-14T07:01:43.831 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.68536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36783.log 2024-09-14T07:01:43.831 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68536.log.gz 2024-09-14T07:01:43.834 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41999.log.gz 2024-09-14T07:01:43.846 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86528.log 2024-09-14T07:01:43.846 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36783.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36783.log.gz 2024-09-14T07:01:43.846 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72399.log.gz 2024-09-14T07:01:43.846 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.70065.log 2024-09-14T07:01:43.846 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47823.log 2024-09-14T07:01:43.847 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70065.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70065.log.gz 2024-09-14T07:01:43.847 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86360.log 2024-09-14T07:01:43.847 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65478.log 2024-09-14T07:01:43.847 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86360.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86528.log.gz 2024-09-14T07:01:43.848 INFO:teuthology.orchestra.run.smithi170.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.86360.log.gz 2024-09-14T07:01:43.848 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42046.log 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65478.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65478.log.gz 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45197.log 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58106.log 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45197.log: /var/log/ceph/cluster1-client.mirror.42046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90075.log 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81922.log 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90075.log: /var/log/ceph/cluster2-client.mirror.58106.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58106.log.gz 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90075.log.gz 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38414.log 2024-09-14T07:01:43.853 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72763.log 2024-09-14T07:01:43.855 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81922.log: /var/log/ceph/cluster1-client.admin.38414.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81922.log.gz 2024-09-14T07:01:43.855 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38414.log.gz 2024-09-14T07:01:43.855 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42537.log 2024-09-14T07:01:43.858 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.72763.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72763.log.gz 2024-09-14T07:01:43.858 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47445.log 2024-09-14T07:01:43.859 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42537.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71928.log.gz 2024-09-14T07:01:43.859 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44337.log 2024-09-14T07:01:43.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62611.log 2024-09-14T07:01:43.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.44337.log: /var/log/ceph/cluster2-client.mirror.44178.log 2024-09-14T07:01:43.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89431.log 2024-09-14T07:01:43.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44178.log: /var/log/ceph/cluster2-client.mirror.62611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39488.log 2024-09-14T07:01:43.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89773.log 2024-09-14T07:01:43.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39488.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62611.log.gz 2024-09-14T07:01:43.864 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83690.log 2024-09-14T07:01:43.864 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36995.log 2024-09-14T07:01:43.864 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3%/var/log/ceph/cluster2-client.mirror.83690.log: -- replaced with /var/log/ceph/cluster2-client.mirror.89431.log.gz/var/log/ceph/cluster1-client.mirror.89773.log: 2024-09-14T07:01:43.864 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83690.log.gz 2024-09-14T07:01:43.864 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89773.log.gz 2024-09-14T07:01:43.867 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74384.log 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74364.log 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.74384.log: /var/log/ceph/cluster2-client.admin.36995.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36995.log.gz 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51573.log 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.74364.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44178.log.gz 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74384.log.gz 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42429.log 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.51573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75667.log 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42429.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74364.log.gz 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51573.log.gz 2024-09-14T07:01:43.868 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74272.log 2024-09-14T07:01:43.918 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80502.log 2024-09-14T07:01:43.918 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.74272.log: /var/log/ceph/cluster1-client.mirror.75667.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74272.log.gz 2024-09-14T07:01:43.918 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75667.log.gz 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42068.log 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50082.log 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42068.log: /var/log/ceph/cluster2-client.mirror.80502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65498.log 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83739.log 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50082.log.gz 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.65498.log: -- replaced with /var/log/ceph/cluster2-client.mirror.80502.log.gz 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74020.log.gz 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.77220.log 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41760.log 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80089.log 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83739.log.gz 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.41760.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65498.log.gz 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77220.log.gzgzip 2024-09-14T07:01:43.919 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65328.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84598.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80089.log.gz 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45309.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65328.log.gz 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79392.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86774.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45309.log: /var/log/ceph/cluster1-client.mirror.79392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79263.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63921.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84598.log.gz 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.79392.log.gz 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86774.log: /var/log/ceph/cluster2-client.mirror.79263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64232.log 2024-09-14T07:01:43.920 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86774.log.gz 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79263.log.gz/var/log/ceph/cluster1-client.mirror.63921.log: 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88463.log 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63921.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69586.log 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.64232.log: /var/log/ceph/cluster1-client.mirror.88463.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64232.log.gz 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88463.log.gz 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84356.log 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44461.log 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84356.log: /var/log/ceph/cluster2-client.mirror.69586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86240.log 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.44461.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69586.log.gz 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84356.log.gz 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74935.log 2024-09-14T07:01:43.921 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50439.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86240.log: /var/log/ceph/cluster2-client.mirror.74935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75164.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38397.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86240.log.gz 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39581.log 87.9% 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.75164.log.gz 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.38397.log: /var/log/ceph/cluster1-client.mirror.89145.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74935.log.gz/var/log/ceph/cluster2-client.mirror.39581.log: 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43547.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44654.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43547.log: /var/log/ceph/cluster1-client.mirror.89145.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89145.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44430.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44654.log: 54.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66432.log 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38397.log.gz 2024-09-14T07:01:43.922 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39581.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.44430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79228.log 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77892.log 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.79228.log: -- /var/log/ceph/cluster2-client.mirror.82942.log 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79228.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77892.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77892.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.71886.log 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66432.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82942.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59260.log 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80888.log 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.59260.log: /var/log/ceph/cluster2-client.mirror.71886.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59260.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45798.log 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80888.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80888.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82942.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71886.log.gz 2024-09-14T07:01:43.923 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43547.log.gz 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80534.log 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36781.log 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44654.log.gz 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45798.log: /var/log/ceph/cluster1-client.mirror.80534.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80534.log.gz 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster1-client.mirror.92187.log 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77581.log 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92187.log: /var/log/ceph/cluster2-client.admin.36781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92187.log.gz 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36781.log.gz 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91583.log 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81874.log 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91583.log: /var/log/ceph/cluster1-client.mirror.77581.log: 87.4% 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77581.log.gz 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91583.log.gz 2024-09-14T07:01:43.924 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79437.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38295.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.79437.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81874.log.gz 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70778.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71097.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87698.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.71097.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71097.log.gz 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr: 53.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38295.log.gz 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.79437.log.gz 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42721.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83852.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% 95.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87698.log.gz 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58809.log.gz 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82822.log 2024-09-14T07:01:43.925 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.42721.log: /var/log/ceph/cluster2-client.mirror.83852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37625.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82822.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82822.log.gz 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52046.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83852.log.gz 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41819.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37625.log: /var/log/ceph/cluster2-client.mirror.52046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45552.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0%/var/log/ceph/cluster2-client.mirror.41819.log: -- replaced with /var/log/ceph/cluster2-client.admin.37625.log.gz 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90381.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92983.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90381.log: /var/log/ceph/cluster1-client.mirror.45552.log: gzip 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90381.log.gz 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70865.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92983.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92983.log.gz 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87247.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42315.log 2024-09-14T07:01:43.926 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.87247.log: -- replaced with /var/log/ceph/cluster2-client.mirror.70865.log.gz 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87247.log.gz 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83457.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46371.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.42315.log: /var/log/ceph/cluster2-client.mirror.83457.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83457.log.gz 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61536.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48710.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.61536.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42315.log.gz 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61536.log.gz 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46371.log: /var/log/ceph/cluster2-client.mirror.48710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40730.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47970.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66915.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92112.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66915.log: /var/log/ceph/cluster2-client.mirror.47970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85661.log 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92112.log.gz 2024-09-14T07:01:43.927 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66915.log.gz 2024-09-14T07:01:43.928 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87607.log 2024-09-14T07:01:43.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68556.log 2024-09-14T07:01:43.928 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87607.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85661.log.gz 2024-09-14T07:01:43.928 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87607.log.gz 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70778.log.gz 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74685.log 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.68556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86887.log 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.74685.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74685.log.gz 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40422.log 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60523.log 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86887.log.gz/var/log/ceph/cluster2-client.mirror.40422.log: 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68556.log.gz 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89526.log 2024-09-14T07:01:43.945 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.60523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84449.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60523.log.gz 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72651.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89526.log.gz 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91949.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.72651.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84449.log.gz 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72651.log.gz 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81781.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46501.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91949.log: /var/log/ceph/cluster1-client.mirror.81781.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81781.log.gz 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45552.log.gz 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72172.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78749.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.72172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78842.log 2024-09-14T07:01:43.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78749.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72172.log.gz 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78749.log.gz 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87990.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65520.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78842.log.gz/var/log/ceph/cluster1-client.mirror.87990.log: 95.6% 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.81462.log.gz 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87990.log.gz 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45286.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78119.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41087.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78119.log: /var/log/ceph/cluster1-client.mirror.45286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41363.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40422.log.gz 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40572.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41363.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46501.log.gz 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42813.log 2024-09-14T07:01:43.947 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81328.log 2024-09-14T07:01:43.948 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45532.log 2024-09-14T07:01:43.948 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81328.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81328.log.gz 2024-09-14T07:01:43.948 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47689.log 2024-09-14T07:01:43.948 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52645.log 2024-09-14T07:01:43.948 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76399.log 2024-09-14T07:01:43.956 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.52645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87382.log 2024-09-14T07:01:43.959 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76399.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76399.log.gz 2024-09-14T07:01:43.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44109.log 2024-09-14T07:01:43.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87382.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87382.log.gz 2024-09-14T07:01:43.976 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44337.log.gz 2024-09-14T07:01:43.984 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45309.log.gz 2024-09-14T07:01:43.992 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36398.log 2024-09-14T07:01:43.994 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44109.log: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91949.log.gz 2024-09-14T07:01:43.995 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49485.log 2024-09-14T07:01:43.995 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87472.log 2024-09-14T07:01:44.010 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.49485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38278.log 2024-09-14T07:01:44.010 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87472.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87472.log.gz 2024-09-14T07:01:44.013 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45286.log.gz 2024-09-14T07:01:44.014 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42046.log.gz 2024-09-14T07:01:44.025 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43020.log 2024-09-14T07:01:44.028 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.38278.log: 69.8% -- replaced with /var/log/ceph/cluster2-client.admin.38278.log.gz 2024-09-14T07:01:44.028 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45798.log.gz 2024-09-14T07:01:44.033 INFO:teuthology.orchestra.run.smithi170.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42429.log.gz 2024-09-14T07:01:44.041 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84327.log 2024-09-14T07:01:44.045 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43020.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43020.log.gz 2024-09-14T07:01:44.065 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39488.log.gz 2024-09-14T07:01:44.067 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40833.log 2024-09-14T07:01:44.070 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84327.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84327.log.gz 2024-09-14T07:01:44.075 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46371.log.gz 2024-09-14T07:01:44.076 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39453.log 2024-09-14T07:01:44.081 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40833.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.41760.log.gz 2024-09-14T07:01:44.081 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45126.log.gz 2024-09-14T07:01:44.081 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45197.log.gz 2024-09-14T07:01:44.082 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42721.log.gz 2024-09-14T07:01:44.089 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40833.log.gz 2024-09-14T07:01:44.089 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53710.log.gz 2024-09-14T07:01:44.090 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90845.log 2024-09-14T07:01:44.091 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90330.log 2024-09-14T07:01:44.092 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90845.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39453.log.gz 2024-09-14T07:01:44.094 INFO:teuthology.orchestra.run.smithi170.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90845.log.gz 2024-09-14T07:01:44.094 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39546.log 2024-09-14T07:01:44.104 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90330.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65520.log.gz 2024-09-14T07:01:44.105 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85985.log 2024-09-14T07:01:44.107 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39546.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39546.log.gz 2024-09-14T07:01:44.109 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45532.log.gz 2024-09-14T07:01:44.117 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37541.log 2024-09-14T07:01:44.119 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85985.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85985.log.gz 2024-09-14T07:01:44.120 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80181.log 2024-09-14T07:01:44.120 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37541.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37541.log.gz 2024-09-14T07:01:44.121 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92137.log 2024-09-14T07:01:44.123 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80181.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80181.log.gz 2024-09-14T07:01:44.123 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80853.log 2024-09-14T07:01:44.123 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92137.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92137.log.gz 2024-09-14T07:01:44.125 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52955.log 2024-09-14T07:01:44.125 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80853.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80853.log.gz 2024-09-14T07:01:44.126 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70357.log 2024-09-14T07:01:44.132 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.52955.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52645.log.gz 2024-09-14T07:01:44.133 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91542.log 2024-09-14T07:01:44.135 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70357.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78119.log.gz 2024-09-14T07:01:44.135 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46644.log 2024-09-14T07:01:44.135 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70357.log.gz 2024-09-14T07:01:44.152 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91542.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50439.log.gz 2024-09-14T07:01:44.152 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79417.log 2024-09-14T07:01:44.154 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45951.log 2024-09-14T07:01:44.154 INFO:teuthology.orchestra.run.smithi170.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46644.log.gz 2024-09-14T07:01:44.157 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79417.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79417.log.gz 2024-09-14T07:01:44.157 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49963.log 2024-09-14T07:01:44.158 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85590.log 2024-09-14T07:01:44.158 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43675.log 2024-09-14T07:01:44.160 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85590.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85590.log.gz 2024-09-14T07:01:44.160 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77195.log 2024-09-14T07:01:44.160 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32796.log 2024-09-14T07:01:44.161 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.77195.log: 87.9%/var/log/ceph/cluster2-client.mirror.49963.log: -- replaced with /var/log/ceph/cluster1-client.mirror.77195.log.gz 2024-09-14T07:01:44.161 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87923.log 2024-09-14T07:01:44.161 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.32796.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32796.log.gz 2024-09-14T07:01:44.164 INFO:teuthology.orchestra.run.smithi170.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43675.log.gz 2024-09-14T07:01:44.164 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63090.log 2024-09-14T07:01:44.164 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87923.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87923.log.gz 2024-09-14T07:01:44.164 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80124.log 2024-09-14T07:01:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63090.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63090.log.gz 2024-09-14T07:01:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75542.log 2024-09-14T07:01:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80124.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80124.log.gz 2024-09-14T07:01:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40515.log 2024-09-14T07:01:44.168 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75542.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75542.log.gz 2024-09-14T07:01:44.168 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68125.log 2024-09-14T07:01:44.169 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88710.log 2024-09-14T07:01:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.68125.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68125.log.gz 2024-09-14T07:01:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41887.log 2024-09-14T07:01:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88710.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88710.log.gz 2024-09-14T07:01:44.171 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44359.log 2024-09-14T07:01:44.173 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91060.log 2024-09-14T07:01:44.173 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70377.log 2024-09-14T07:01:44.173 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91060.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91060.log.gz 2024-09-14T07:01:44.174 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66452.log 2024-09-14T07:01:44.175 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.70377.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40515.log.gz 2024-09-14T07:01:44.177 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41887.log.gz 2024-09-14T07:01:44.177 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70377.log.gz 2024-09-14T07:01:44.177 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42609.log 2024-09-14T07:01:44.180 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.66452.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66452.log.gz 2024-09-14T07:01:44.180 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90486.log 2024-09-14T07:01:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88418.log 2024-09-14T07:01:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90486.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90486.log.gz 2024-09-14T07:01:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92262.log 2024-09-14T07:01:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88418.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88418.log.gz 2024-09-14T07:01:44.183 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83173.log 2024-09-14T07:01:44.184 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92262.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92262.log.gz 2024-09-14T07:01:44.184 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46459.log 2024-09-14T07:01:44.187 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83173.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83173.log.gz 2024-09-14T07:01:44.187 INFO:teuthology.orchestra.run.smithi170.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42609.log.gz 2024-09-14T07:01:44.187 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38516.log 2024-09-14T07:01:44.187 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64544.log 2024-09-14T07:01:44.188 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88373.log 2024-09-14T07:01:44.188 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.44461.log.gz 2024-09-14T07:01:44.190 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.64544.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64544.log.gz 2024-09-14T07:01:44.190 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92647.log 2024-09-14T07:01:44.193 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88373.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88373.log.gz 2024-09-14T07:01:44.193 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86149.log 2024-09-14T07:01:44.193 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92647.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92647.log.gz 2024-09-14T07:01:44.193 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90569.log 2024-09-14T07:01:44.194 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86149.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86149.log.gz 2024-09-14T07:01:44.194 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46459.log.gz 2024-09-14T07:01:44.200 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42387.log 2024-09-14T07:01:44.203 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90569.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90569.log.gz 2024-09-14T07:01:44.205 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89379.log 2024-09-14T07:01:44.205 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85956.log 2024-09-14T07:01:44.205 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89379.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89379.log.gz 2024-09-14T07:01:44.206 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88508.log 2024-09-14T07:01:44.209 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85956.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85956.log.gz 2024-09-14T07:01:44.209 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46084.log 2024-09-14T07:01:44.210 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88508.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88508.log.gz 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.36398.log.gz 2024-09-14T07:01:44.210 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:44.210 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38528.log 2024-09-14T07:01:44.211 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.46084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65453.log 2024-09-14T07:01:44.212 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.38528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67654.log 2024-09-14T07:01:44.213 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.65453.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65453.log.gz 2024-09-14T07:01:44.213 INFO:teuthology.orchestra.run.smithi170.stderr: 19.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38528.log.gz 2024-09-14T07:01:44.215 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92857.log 2024-09-14T07:01:44.217 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.67654.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67654.log.gz 2024-09-14T07:01:44.217 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37331.log 2024-09-14T07:01:44.220 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92857.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92857.log.gz 2024-09-14T07:01:44.220 INFO:teuthology.orchestra.run.smithi170.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57703.log.gz 2024-09-14T07:01:44.220 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46564.log 2024-09-14T07:01:44.220 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.37331.log: -- /var/log/ceph/cluster2-client.mirror.51843.log 2024-09-14T07:01:44.220 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37331.log.gz 2024-09-14T07:01:44.220 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48074.log 2024-09-14T07:01:44.220 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90222.log 2024-09-14T07:01:44.223 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48074.log: /var/log/ceph/cluster2-client.mirror.51843.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51843.log.gz 2024-09-14T07:01:44.223 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77648.log 2024-09-14T07:01:44.223 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50662.log 2024-09-14T07:01:44.223 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77648.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90222.log.gz 2024-09-14T07:01:44.223 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49940.log 2024-09-14T07:01:44.224 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50662.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50662.log.gz 2024-09-14T07:01:44.225 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92731.log 2024-09-14T07:01:44.225 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.49940.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49940.log.gz 2024-09-14T07:01:44.228 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77128.log 2024-09-14T07:01:44.229 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92731.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92731.log.gz 2024-09-14T07:01:44.229 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83408.log 2024-09-14T07:01:44.229 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77128.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77128.log.gz 2024-09-14T07:01:44.229 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87584.log 2024-09-14T07:01:44.231 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83408.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83408.log.gz 2024-09-14T07:01:44.231 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90507.log 2024-09-14T07:01:44.231 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87584.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87584.log.gz 2024-09-14T07:01:44.231 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76098.log 2024-09-14T07:01:44.233 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90507.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90507.log.gz 2024-09-14T07:01:44.233 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85865.log 2024-09-14T07:01:44.235 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.76098.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76098.log.gz 2024-09-14T07:01:44.235 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85175.log 2024-09-14T07:01:44.236 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85865.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85865.log.gz 2024-09-14T07:01:44.236 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52023.log 2024-09-14T07:01:44.239 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85175.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85175.log.gz 2024-09-14T07:01:44.239 INFO:teuthology.orchestra.run.smithi170.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49485.log.gz 2024-09-14T07:01:44.239 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38363.log 2024-09-14T07:01:44.240 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88553.log 2024-09-14T07:01:44.240 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.52023.log: /var/log/ceph/cluster2-client.mirror.38363.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52023.log.gz 2024-09-14T07:01:44.240 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89052.log 2024-09-14T07:01:44.240 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88553.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88553.log.gz 2024-09-14T07:01:44.242 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68170.log 2024-09-14T07:01:44.243 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89052.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38363.log.gz 2024-09-14T07:01:44.243 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89052.log.gz 2024-09-14T07:01:44.243 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60546.log 2024-09-14T07:01:44.243 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88867.log 2024-09-14T07:01:44.244 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.60546.log: /var/log/ceph/cluster1-client.mirror.68170.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68170.log.gz 2024-09-14T07:01:44.244 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73978.log 2024-09-14T07:01:44.246 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.88867.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88867.log.gz 2024-09-14T07:01:44.246 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80156.log 2024-09-14T07:01:44.248 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73978.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73978.log.gz 2024-09-14T07:01:44.248 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37163.log 2024-09-14T07:01:44.249 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.80156.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80156.log.gz 2024-09-14T07:01:44.250 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86932.log 2024-09-14T07:01:44.250 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37163.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37163.log.gz 2024-09-14T07:01:44.251 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81420.log 2024-09-14T07:01:44.251 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86932.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86932.log.gz 2024-09-14T07:01:44.252 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40957.log 2024-09-14T07:01:44.254 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81420.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81420.log.gz 2024-09-14T07:01:44.254 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46733.log 2024-09-14T07:01:44.255 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.40957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42768.log 2024-09-14T07:01:44.260 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42509.log 2024-09-14T07:01:44.261 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.42768.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44912.log 2024-09-14T07:01:44.264 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.42509.log: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44359.log.gz 2024-09-14T07:01:44.265 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76364.log 2024-09-14T07:01:44.265 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44912.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77648.log.gz 2024-09-14T07:01:44.266 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69089.log 2024-09-14T07:01:44.266 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76364.log.gz 2024-09-14T07:01:44.268 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.69089.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69089.log.gz 2024-09-14T07:01:44.268 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76842.log 2024-09-14T07:01:44.268 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93122.log 2024-09-14T07:01:44.268 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56540.log 2024-09-14T07:01:44.269 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76842.log.gz 2024-09-14T07:01:44.270 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.93122.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93122.log.gz 2024-09-14T07:01:44.271 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.56540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42701.log 2024-09-14T07:01:44.271 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56540.log.gz 2024-09-14T07:01:44.272 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43113.log 2024-09-14T07:01:44.274 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46890.log 2024-09-14T07:01:44.274 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83244.log 2024-09-14T07:01:44.274 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.46890.log: -- /var/log/ceph/cluster2-client.mirror.72785.log 2024-09-14T07:01:44.274 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40618.log 2024-09-14T07:01:44.275 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83244.log: /var/log/ceph/cluster2-client.mirror.72785.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83244.log.gz 2024-09-14T07:01:44.276 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58371.log 2024-09-14T07:01:44.278 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40618.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.42509.log.gz 2024-09-14T07:01:44.278 INFO:teuthology.orchestra.run.smithi170.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43113.log.gz 2024-09-14T07:01:44.279 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69746.log 2024-09-14T07:01:44.288 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.58371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65869.log 2024-09-14T07:01:44.295 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.69746.log: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92941.log 2024-09-14T07:01:44.295 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69746.log.gz 2024-09-14T07:01:44.308 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65869.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65869.log.gz 2024-09-14T07:01:44.314 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45951.log.gz 2024-09-14T07:01:44.315 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60865.log 2024-09-14T07:01:44.315 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92941.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92941.log.gz 2024-09-14T07:01:44.328 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91233.log 2024-09-14T07:01:44.338 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.60865.log: 88.0% 89.0% -- replaced with /var/log/ceph/cluster1-client.admin.44430.log.gz 2024-09-14T07:01:44.338 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60865.log.gz 2024-09-14T07:01:44.346 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84469.log 2024-09-14T07:01:44.351 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91233.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52955.log.gz 2024-09-14T07:01:44.352 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42701.log.gz 2024-09-14T07:01:44.352 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50105.log 2024-09-14T07:01:44.355 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84469.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84469.log.gz 2024-09-14T07:01:44.355 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69771.log 2024-09-14T07:01:44.355 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50105.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46564.log.gz 2024-09-14T07:01:44.365 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40671.log 2024-09-14T07:01:44.366 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69771.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69771.log.gz 2024-09-14T07:01:44.379 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90974.log 2024-09-14T07:01:44.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.40671.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.40671.log.gz 2024-09-14T07:01:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75912.log 2024-09-14T07:01:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90974.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90974.log.gz 2024-09-14T07:01:44.410 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44689.log 2024-09-14T07:01:44.411 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.75912.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75912.log.gz 2024-09-14T07:01:44.418 INFO:teuthology.orchestra.run.smithi170.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52046.log.gz 2024-09-14T07:01:44.419 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72785.log.gz 2024-09-14T07:01:44.425 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36657.log 2024-09-14T07:01:44.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81084.log 2024-09-14T07:01:44.425 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36657.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36657.log.gz 2024-09-14T07:01:44.425 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83832.log 2024-09-14T07:01:44.436 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81084.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44689.log.gz 2024-09-14T07:01:44.437 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45442.log 2024-09-14T07:01:44.438 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83832.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83832.log.gz 2024-09-14T07:01:44.443 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90304.log 2024-09-14T07:01:44.445 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45442.log: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44912.log.gz 2024-09-14T07:01:44.447 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49963.log.gz 2024-09-14T07:01:44.448 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89207.log 2024-09-14T07:01:44.449 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90304.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90304.log.gz 2024-09-14T07:01:44.449 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35774.log 2024-09-14T07:01:44.453 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89207.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89207.log.gz 2024-09-14T07:01:44.453 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37457.log 2024-09-14T07:01:44.453 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.35774.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35774.log.gz 2024-09-14T07:01:44.454 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45888.log 2024-09-14T07:01:44.454 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37457.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37457.log.gz 2024-09-14T07:01:44.455 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84143.log 2024-09-14T07:01:44.455 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63125.log 2024-09-14T07:01:44.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84143.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84143.log.gz 2024-09-14T07:01:44.457 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70444.log 2024-09-14T07:01:44.458 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63125.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63125.log.gz 2024-09-14T07:01:44.458 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91026.log 2024-09-14T07:01:44.459 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70444.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70444.log.gz 2024-09-14T07:01:44.459 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92043.log 2024-09-14T07:01:44.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91026.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91026.log.gz 2024-09-14T07:01:44.462 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72300.log 2024-09-14T07:01:44.462 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92043.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92043.log.gz 2024-09-14T07:01:44.464 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67293.log 2024-09-14T07:01:44.464 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.72300.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72300.log.gz 2024-09-14T07:01:44.464 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88688.log 2024-09-14T07:01:44.468 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67293.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67293.log.gz 2024-09-14T07:01:44.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63424.log 2024-09-14T07:01:44.468 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88688.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88688.log.gz 2024-09-14T07:01:44.468 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78463.log 2024-09-14T07:01:44.480 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61411.log 2024-09-14T07:01:44.485 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78463.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78463.log.gz 2024-09-14T07:01:44.485 INFO:teuthology.orchestra.run.smithi170.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.90330.log.gz 2024-09-14T07:01:44.485 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55122.log 2024-09-14T07:01:44.486 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61411.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61411.log.gz 2024-09-14T07:01:44.486 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87022.log 2024-09-14T07:01:44.505 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.55122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46153.log 2024-09-14T07:01:44.506 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87022.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87022.log.gz 2024-09-14T07:01:44.510 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46084.log.gz 2024-09-14T07:01:44.515 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81084.log.gz 2024-09-14T07:01:44.515 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75196.log 2024-09-14T07:01:44.515 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.46153.log: -- /var/log/ceph/cluster2-client.mirror.53687.log 2024-09-14T07:01:44.517 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.75196.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75196.log.gz 2024-09-14T07:01:44.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88103.log 2024-09-14T07:01:44.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.53687.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53687.log.gz 2024-09-14T07:01:44.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80559.log 2024-09-14T07:01:44.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88103.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88103.log.gz 2024-09-14T07:01:44.519 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92899.log 2024-09-14T07:01:44.522 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80559.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.80559.log.gz 2024-09-14T07:01:44.522 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38106.log 2024-09-14T07:01:44.522 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92899.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92899.log.gz 2024-09-14T07:01:44.522 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-14T07:01:44.523 INFO:teuthology.orchestra.run.smithi001.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-14T07:01:44.526 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82145.log 2024-09-14T07:01:44.526 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43454.log 2024-09-14T07:01:44.526 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46153.log.gz 2024-09-14T07:01:44.526 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82145.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82145.log.gz 2024-09-14T07:01:44.526 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77084.log 2024-09-14T07:01:44.527 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82449.log 2024-09-14T07:01:44.531 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77084.log: 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38106.log.gz 2024-09-14T07:01:44.532 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43454.log.gz 2024-09-14T07:01:44.545 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45888.log.gz 2024-09-14T07:01:44.559 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82165.log 2024-09-14T07:01:44.561 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82449.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82449.log.gz 2024-09-14T07:01:44.569 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91233.log.gz 2024-09-14T07:01:44.569 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36909.log 2024-09-14T07:01:44.571 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82165.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82165.log.gz 2024-09-14T07:01:44.573 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86684.log 2024-09-14T07:01:44.573 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36909.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36909.log.gz 2024-09-14T07:01:44.574 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41340.log 2024-09-14T07:01:44.574 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86684.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86684.log.gz 2024-09-14T07:01:44.575 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86289.log 2024-09-14T07:01:44.575 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64715.log 2024-09-14T07:01:44.576 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86289.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86289.log.gz 2024-09-14T07:01:44.577 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38174.log 2024-09-14T07:01:44.577 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45442.log.gz 2024-09-14T07:01:44.578 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.64715.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64715.log.gz 2024-09-14T07:01:44.578 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43396.log 2024-09-14T07:01:44.578 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74810.log 2024-09-14T07:01:44.579 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58786.log 2024-09-14T07:01:44.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.74810.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74810.log.gz 2024-09-14T07:01:44.581 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66247.log 2024-09-14T07:01:44.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.58786.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58786.log.gz 2024-09-14T07:01:44.582 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37079.log 2024-09-14T07:01:44.584 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66247.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66247.log.gz 2024-09-14T07:01:44.584 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86056.log 2024-09-14T07:01:44.584 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37079.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37079.log.gz 2024-09-14T07:01:44.585 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83286.log 2024-09-14T07:01:44.586 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86056.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86056.log.gz 2024-09-14T07:01:44.586 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44290.log 2024-09-14T07:01:44.589 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83286.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83286.log.gz 2024-09-14T07:01:44.589 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87134.log 2024-09-14T07:01:44.596 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.44290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37709.log 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38174.log.gz 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87134.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87134.log.gz 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68093.log 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43396.log.gz 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91542.log.gz 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47800.log 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37709.log.gz/var/log/ceph/cluster2-client.mirror.68093.log: 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68093.log.gz 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.68857.log 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.47800.log: -- /var/log/ceph/cluster2-client.mirror.86331.log 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42790.log 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86331.log: /var/log/ceph/cluster2-client.mirror.68857.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68857.log.gz 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63802.log 2024-09-14T07:01:44.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42790.log: gzip -5 --verbose 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86331.log.gz 2024-09-14T07:01:44.598 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster1-client.mirror.89800.log 2024-09-14T07:01:44.598 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87539.log 2024-09-14T07:01:44.603 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89800.log: /var/log/ceph/cluster2-client.mirror.63802.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89800.log.gz 2024-09-14T07:01:44.603 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86078.log 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87539.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87539.log.gz 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38380.log 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86078.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.86078.log.gz 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38346.log 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66623.log 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46021.log 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66623.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66623.log.gz 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56657.log 2024-09-14T07:01:44.604 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46710.log 2024-09-14T07:01:44.606 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.56657.log: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63802.log.gz 2024-09-14T07:01:44.606 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66407.log 2024-09-14T07:01:44.607 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38329.log 2024-09-14T07:01:44.608 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.66407.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66407.log.gz 2024-09-14T07:01:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45243.log 2024-09-14T07:01:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73482.log 2024-09-14T07:01:44.612 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80579.log 2024-09-14T07:01:44.616 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.73482.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73482.log.gz 2024-09-14T07:01:44.616 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86382.log 2024-09-14T07:01:44.621 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.80579.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80579.log.gz 2024-09-14T07:01:44.621 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67721.log 2024-09-14T07:01:44.622 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86382.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86382.log.gz 2024-09-14T07:01:44.622 INFO:teuthology.orchestra.run.smithi170.stderr: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38380.log.gz 2024-09-14T07:01:44.625 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32551.log 2024-09-14T07:01:44.631 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67721.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58371.log.gz 2024-09-14T07:01:44.633 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60546.log.gz 2024-09-14T07:01:44.636 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77084.log.gz 2024-09-14T07:01:44.647 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39395.log 2024-09-14T07:01:44.647 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.32551.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64653.log 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.32551.log.gz 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89247.log 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39395.log: /var/log/ceph/cluster2-client.mirror.64653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62233.log 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64653.log.gz 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54916.log 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91790.log 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62233.log.gz 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.54916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36533.log 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91790.log: 87.3% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91790.log.gz 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.89247.log.gz 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82729.log 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36533.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36533.log.gz 2024-09-14T07:01:44.648 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86909.log 2024-09-14T07:01:44.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85146.log 2024-09-14T07:01:44.649 INFO:teuthology.orchestra.run.smithi170.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.82729.log.gz 2024-09-14T07:01:44.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86909.log: gzip -5 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86909.log.gz 2024-09-14T07:01:44.649 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.63854.log 2024-09-14T07:01:44.649 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88643.log 2024-09-14T07:01:44.649 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63854.log: /var/log/ceph/cluster2-client.mirror.85146.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63854.log.gz 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85146.log.gz 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38191.log 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88643.log: 87.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77240.log 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.88643.log.gz 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.38191.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38191.log.gz 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90548.log 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.77240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40050.log 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90548.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77240.log.gz 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73921.log 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38438.log 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73921.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90548.log.gz 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73921.log.gz 2024-09-14T07:01:44.658 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38472.log 2024-09-14T07:01:44.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.38438.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36491.log 2024-09-14T07:01:44.659 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.38438.log.gz 2024-09-14T07:01:44.659 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78784.log 2024-09-14T07:01:44.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38472.log: /var/log/ceph/cluster2-client.admin.36491.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36491.log.gz 2024-09-14T07:01:44.659 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.39081.log 2024-09-14T07:01:44.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71142.log 2024-09-14T07:01:44.659 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78784.log.gz 2024-09-14T07:01:44.665 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39081.log: gzip -5 --verbose -- 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38329.log.gz 2024-09-14T07:01:44.666 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.74778.log 2024-09-14T07:01:44.666 INFO:teuthology.orchestra.run.smithi170.stderr: 63.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38346.log.gz 2024-09-14T07:01:44.666 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.71142.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40050.log.gz 2024-09-14T07:01:44.667 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39081.log.gz 2024-09-14T07:01:44.667 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71142.log.gz 2024-09-14T07:01:44.670 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39395.log.gz 2024-09-14T07:01:44.670 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40618.log.gz 2024-09-14T07:01:44.670 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61469.log 2024-09-14T07:01:44.670 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62367.log 2024-09-14T07:01:44.670 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61469.log: /var/log/ceph/cluster2-client.mirror.74778.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61469.log.gz 2024-09-14T07:01:44.670 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74778.log.gz 2024-09-14T07:01:44.670 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78877.log 2024-09-14T07:01:44.671 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41251.log 2024-09-14T07:01:44.674 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78877.log: /var/log/ceph/cluster2-client.mirror.62367.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78877.log.gz 2024-09-14T07:01:44.674 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41954.log 2024-09-14T07:01:44.675 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36373.log 2024-09-14T07:01:44.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.41954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39674.log 2024-09-14T07:01:44.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36373.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36373.log.gz 2024-09-14T07:01:44.684 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38038.log 2024-09-14T07:01:44.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40910.log 2024-09-14T07:01:44.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.38038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90201.log 2024-09-14T07:01:44.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.40910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84185.log 2024-09-14T07:01:44.687 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90201.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90201.log.gz 2024-09-14T07:01:44.687 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81017.log 2024-09-14T07:01:44.693 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84185.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84185.log.gz 2024-09-14T07:01:44.695 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41954.log.gz 2024-09-14T07:01:44.697 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82771.log 2024-09-14T07:01:44.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.81017.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81017.log.gz 2024-09-14T07:01:44.699 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39674.log.gz 2024-09-14T07:01:44.700 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50916.log 2024-09-14T07:01:44.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82771.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82771.log.gz 2024-09-14T07:01:44.703 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92064.log 2024-09-14T07:01:44.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50916.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50916.log.gz 2024-09-14T07:01:44.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71243.log 2024-09-14T07:01:44.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92064.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92064.log.gz 2024-09-14T07:01:44.705 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82378.log 2024-09-14T07:01:44.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.71243.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71243.log.gz 2024-09-14T07:01:44.707 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68000.log 2024-09-14T07:01:44.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82378.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.38038.log.gz 2024-09-14T07:01:44.709 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82378.log.gz 2024-09-14T07:01:44.709 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39209.log 2024-09-14T07:01:44.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.68000.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68000.log.gz 2024-09-14T07:01:44.711 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40957.log.gz 2024-09-14T07:01:44.712 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91299.log 2024-09-14T07:01:44.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41452.log 2024-09-14T07:01:44.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91299.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.91299.log.gz 2024-09-14T07:01:44.715 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42889.log 2024-09-14T07:01:44.715 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86999.log 2024-09-14T07:01:44.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.42889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91454.log 2024-09-14T07:01:44.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86999.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86999.log.gz 2024-09-14T07:01:44.719 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78483.log 2024-09-14T07:01:44.720 INFO:teuthology.orchestra.run.smithi001.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-14T07:01:44.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91454.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91454.log.gz 2024-09-14T07:01:44.722 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72357.log 2024-09-14T07:01:44.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.78483.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78483.log.gz 2024-09-14T07:01:44.727 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45818.log 2024-09-14T07:01:44.727 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63424.log.gz 2024-09-14T07:01:44.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.72357.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72357.log.gz 2024-09-14T07:01:44.728 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90804.log 2024-09-14T07:01:44.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45818.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27719.log 2024-09-14T07:01:44.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90804.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90804.log.gz 2024-09-14T07:01:44.732 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64267.log 2024-09-14T07:01:44.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.tmp-client.admin.27719.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27719.log.gz 2024-09-14T07:01:44.732 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63503.log 2024-09-14T07:01:44.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.64267.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64267.log.gz 2024-09-14T07:01:44.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63503.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63503.log.gz 2024-09-14T07:01:44.735 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69991.log 2024-09-14T07:01:44.736 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45329.log 2024-09-14T07:01:44.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.69991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47422.log 2024-09-14T07:01:44.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48963.log 2024-09-14T07:01:44.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83974.log 2024-09-14T07:01:44.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44766.log 2024-09-14T07:01:44.740 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83974.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83974.log.gz 2024-09-14T07:01:44.740 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91812.log 2024-09-14T07:01:44.740 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69991.log.gz 2024-09-14T07:01:44.744 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85936.log 2024-09-14T07:01:44.744 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91812.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91812.log.gz 2024-09-14T07:01:44.745 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91989.log 2024-09-14T07:01:44.745 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85936.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85936.log.gz 2024-09-14T07:01:44.745 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42407.log 2024-09-14T07:01:44.748 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91989.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.91989.log.gz 2024-09-14T07:01:44.750 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85033.log 2024-09-14T07:01:44.750 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.42407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73415.log 2024-09-14T07:01:44.752 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85033.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46021.log.gz 2024-09-14T07:01:44.752 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39860.log 2024-09-14T07:01:44.752 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73415.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85033.log.gz 2024-09-14T07:01:44.752 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73415.log.gz 2024-09-14T07:01:44.753 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84691.log 2024-09-14T07:01:44.754 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66474.log 2024-09-14T07:01:44.754 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84691.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84691.log.gz 2024-09-14T07:01:44.755 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45462.log 2024-09-14T07:01:44.759 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66474.log: /var/log/ceph/cluster1-client.mirror.45462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87179.log 2024-09-14T07:01:44.760 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87720.log 2024-09-14T07:01:44.760 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87179.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87179.log.gz 2024-09-14T07:01:44.761 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71542.log 2024-09-14T07:01:44.762 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87720.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87720.log.gz 2024-09-14T07:01:44.764 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66474.log.gz 2024-09-14T07:01:44.764 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.79325.log 2024-09-14T07:01:44.764 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60502.log 2024-09-14T07:01:44.765 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79325.log: /var/log/ceph/cluster2-client.mirror.71542.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79325.log.gz 2024-09-14T07:01:44.765 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78097.log 2024-09-14T07:01:44.765 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.60502.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60502.log.gz 2024-09-14T07:01:44.767 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71594.log 2024-09-14T07:01:44.769 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.78097.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78097.log.gz 2024-09-14T07:01:44.769 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91324.log 2024-09-14T07:01:44.770 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.71594.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71594.log.gz 2024-09-14T07:01:44.771 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42858.log 2024-09-14T07:01:44.779 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91324.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44290.log.gz 2024-09-14T07:01:44.781 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62367.log.gz 2024-09-14T07:01:44.781 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87630.log 2024-09-14T07:01:44.783 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.42858.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39209.log.gz 2024-09-14T07:01:44.785 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62739.log 2024-09-14T07:01:44.785 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87630.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87630.log.gz 2024-09-14T07:01:44.785 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68511.log 2024-09-14T07:01:44.785 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62739.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62739.log.gz 2024-09-14T07:01:44.787 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82964.log 2024-09-14T07:01:44.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.68511.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68511.log.gz 2024-09-14T07:01:44.787 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65293.log 2024-09-14T07:01:44.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82964.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82964.log.gz 2024-09-14T07:01:44.790 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-14T07:01:44.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65293.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65293.log.gz 2024-09-14T07:01:44.790 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91392.log 2024-09-14T07:01:44.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67965.log 2024-09-14T07:01:44.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91392.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91392.log.gz 2024-09-14T07:01:44.793 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48733.log 2024-09-14T07:01:44.793 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67965.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67965.log.gz 2024-09-14T07:01:44.794 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39732.log 2024-09-14T07:01:44.794 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84398.log 2024-09-14T07:01:44.794 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47053.log 2024-09-14T07:01:44.795 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84398.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84398.log.gz 2024-09-14T07:01:44.797 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73886.log 2024-09-14T07:01:44.797 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84165.log 2024-09-14T07:01:44.797 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73886.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73886.log.gz 2024-09-14T07:01:44.798 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85519.log 2024-09-14T07:01:44.798 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84165.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84165.log.gz 2024-09-14T07:01:44.799 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67721.log.gz 2024-09-14T07:01:44.799 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61847.log 2024-09-14T07:01:44.800 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85519.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85519.log.gz 2024-09-14T07:01:44.800 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61581.log 2024-09-14T07:01:44.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61847.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61847.log.gz 2024-09-14T07:01:44.801 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44991.log 2024-09-14T07:01:44.802 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.61581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32730.log 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44823.log 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.32730.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32730.log.gz 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61581.log.gz 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79703.log 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38639.log 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88755.log 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79703.log.gz 2024-09-14T07:01:44.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85197.log 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88755.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88755.log.gz 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.73029.log 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73450.log 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73029.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85197.log.gz 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89903.log 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73450.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73029.log.gz 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73450.log.gz 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr: 95.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91324.log.gz 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45329.log.gz 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78909.log 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40547.log 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89903.log.gz 2024-09-14T07:01:44.811 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.78909.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78909.log.gz 2024-09-14T07:01:44.812 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69621.log 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.40547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38217.log 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/cluster2-client.mirror.69621.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91709.log 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.38217.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69621.log.gz 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38217.log.gz 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56860.log 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87833.log 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91709.log.gz 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.56860.log: gzip -5 --verbose -- 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56860.log.gz 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.75986.log 2024-09-14T07:01:44.819 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61939.log 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87833.log.gz 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75986.log: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54916.log.gz 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75986.log.gz 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90180.log 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86729.log 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91635.log 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90180.log.gz 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63966.log 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86729.log: /var/log/ceph/cluster2-client.mirror.91635.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86729.log.gz 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61939.log.gz 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91635.log.gz 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43877.log 2024-09-14T07:01:44.820 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.63966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82358.log 2024-09-14T07:01:44.821 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.43877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83781.log 2024-09-14T07:01:44.821 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87404.log 2024-09-14T07:01:44.821 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83781.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82358.log.gz 2024-09-14T07:01:44.821 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83781.log.gz 2024-09-14T07:01:44.828 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63966.log.gz 2024-09-14T07:01:44.828 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46216.log 2024-09-14T07:01:44.828 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87404.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87404.log.gz 2024-09-14T07:01:44.828 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster1-client.admin.41424.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46241.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.41424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43299.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65961.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82658.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46195.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68192.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50685.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82658.log.gz 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89077.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50685.log: /var/log/ceph/cluster2-client.mirror.68192.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65961.log.gz 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45352.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88778.log 2024-09-14T07:01:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93025.log 2024-09-14T07:01:44.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88778.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88778.log.gz 2024-09-14T07:01:44.832 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75221.log 2024-09-14T07:01:44.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.93025.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93025.log.gz 2024-09-14T07:01:44.832 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91207.log 2024-09-14T07:01:44.833 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75221.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42768.log.gz 2024-09-14T07:01:44.833 INFO:teuthology.orchestra.run.smithi170.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47800.log.gz 2024-09-14T07:01:44.837 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75221.log.gz 2024-09-14T07:01:44.837 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84782.log 2024-09-14T07:01:44.838 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91207.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91207.log.gz 2024-09-14T07:01:44.838 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42180.log 2024-09-14T07:01:44.842 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84782.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84782.log.gz 2024-09-14T07:01:44.842 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91838.log 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50105.log.gz 2024-09-14T07:01:44.842 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:44.842 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82984.log 2024-09-14T07:01:44.842 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90866.log 2024-09-14T07:01:44.845 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91838.log: /var/log/ceph/cluster2-client.mirror.82984.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82984.log.gz 2024-09-14T07:01:44.845 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61882.log 2024-09-14T07:01:44.845 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90866.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90866.log.gz 2024-09-14T07:01:44.845 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88305.log 2024-09-14T07:01:44.845 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61882.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61882.log.gz 2024-09-14T07:01:44.849 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67199.log 2024-09-14T07:01:44.849 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88305.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88305.log.gz 2024-09-14T07:01:44.849 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52465.log 2024-09-14T07:01:44.849 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67199.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67199.log.gz 2024-09-14T07:01:44.849 INFO:teuthology.orchestra.run.smithi170.stderr: 95.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89077.log.gz 2024-09-14T07:01:44.849 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45994.log 2024-09-14T07:01:44.849 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.52465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39953.log 2024-09-14T07:01:44.853 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66340.log 2024-09-14T07:01:44.853 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83337.log 2024-09-14T07:01:44.853 INFO:teuthology.orchestra.run.smithi170.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56657.log.gz 2024-09-14T07:01:44.853 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66340.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66340.log.gz 2024-09-14T07:01:44.853 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88848.log 2024-09-14T07:01:44.858 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41724.log 2024-09-14T07:01:44.858 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.88848.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83337.log.gz 2024-09-14T07:01:44.858 INFO:teuthology.orchestra.run.smithi170.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88848.log.gz 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86594.log 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86842.log 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67622.log 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86594.log.gz 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81363.log 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67622.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86842.log.gz 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67622.log.gz 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82700.log 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.81363.log: /var/log/ceph/cluster1-client.mirror.88395.log 2024-09-14T07:01:44.859 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81363.log.gz 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82700.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43877.log.gz 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62816.log 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88395.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82700.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88395.log.gz 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62838.log 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.62816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86098.log 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62838.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62816.log.gz 2024-09-14T07:01:44.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68444.log 2024-09-14T07:01:44.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86098.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86098.log.gz 2024-09-14T07:01:44.868 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45618.log 2024-09-14T07:01:44.868 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.68444.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68444.log.gz 2024-09-14T07:01:44.876 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55661.log 2024-09-14T07:01:44.876 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42985.log 2024-09-14T07:01:44.885 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.55661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66957.log 2024-09-14T07:01:44.895 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64618.log 2024-09-14T07:01:44.896 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66957.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71542.log.gz 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37247.log 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.64618.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64618.log.gz 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60798.log 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37247.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37247.log.gz 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46667.log 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.60798.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60798.log.gz 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88013.log 2024-09-14T07:01:44.900 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85448.log 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88013.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88013.log.gz 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37205.log 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39732.log.gz 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85448.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39860.log.gz 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39953.log.gz 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85448.log.gz 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45243.log.gz 2024-09-14T07:01:44.903 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40547.log.gz 2024-09-14T07:01:44.904 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40910.log.gz 2024-09-14T07:01:44.909 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46041.log 2024-09-14T07:01:44.910 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37205.log.gz 2024-09-14T07:01:44.920 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.41424.log.gz 2024-09-14T07:01:44.926 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45974.log 2024-09-14T07:01:44.938 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.46041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91083.log 2024-09-14T07:01:44.962 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90588.log 2024-09-14T07:01:44.963 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91083.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91083.log.gz 2024-09-14T07:01:44.969 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83710.log 2024-09-14T07:01:44.972 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90588.log: 95.8% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90588.log.gz 2024-09-14T07:01:44.972 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55122.log.gz 2024-09-14T07:01:44.972 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88148.log 2024-09-14T07:01:44.974 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83710.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83710.log.gz 2024-09-14T07:01:44.975 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85774.log 2024-09-14T07:01:44.976 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88148.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88148.log.gz 2024-09-14T07:01:44.976 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87359.log 2024-09-14T07:01:44.979 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85774.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85774.log.gz 2024-09-14T07:01:44.979 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81395.log 2024-09-14T07:01:44.981 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87359.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87359.log.gz 2024-09-14T07:01:44.981 INFO:teuthology.orchestra.run.smithi170.stderr: 88.9% 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42889.log.gz 2024-09-14T07:01:44.981 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.42858.log.gz 2024-09-14T07:01:44.981 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70830.log 2024-09-14T07:01:44.981 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.81395.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81395.log.gz 2024-09-14T07:01:44.983 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88193.log 2024-09-14T07:01:44.983 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70830.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70830.log.gz 2024-09-14T07:01:44.983 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83903.log 2024-09-14T07:01:44.984 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88193.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88193.log.gz 2024-09-14T07:01:44.984 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86662.log 2024-09-14T07:01:44.986 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83903.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83903.log.gz 2024-09-14T07:01:44.986 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83499.log 2024-09-14T07:01:44.986 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86662.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86662.log.gz 2024-09-14T07:01:44.987 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39825.log 2024-09-14T07:01:44.990 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83499.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83499.log.gz 2024-09-14T07:01:44.990 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67268.log 2024-09-14T07:01:44.991 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45818.log.gz 2024-09-14T07:01:44.991 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45084.log 2024-09-14T07:01:44.991 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.67268.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67268.log.gz 2024-09-14T07:01:44.992 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42790.log.gz 2024-09-14T07:01:44.993 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36575.log 2024-09-14T07:01:44.994 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77549.log 2024-09-14T07:01:44.995 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36575.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36575.log.gz 2024-09-14T07:01:44.995 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37415.log 2024-09-14T07:01:45.000 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77549.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77549.log.gz 2024-09-14T07:01:45.000 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67234.log 2024-09-14T07:01:45.000 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37415.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37415.log.gz 2024-09-14T07:01:45.004 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44382.log 2024-09-14T07:01:45.005 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67234.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67234.log.gz 2024-09-14T07:01:45.008 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92022.log 2024-09-14T07:01:45.011 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71906.log 2024-09-14T07:01:45.011 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92022.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92022.log.gz 2024-09-14T07:01:45.013 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90528.log 2024-09-14T07:01:45.014 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.71906.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71906.log.gz 2024-09-14T07:01:45.017 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89295.log 2024-09-14T07:01:45.019 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90528.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66957.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90528.log.gz 2024-09-14T07:01:45.019 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:45.019 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91104.log 2024-09-14T07:01:45.020 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85126.log 2024-09-14T07:01:45.020 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91104.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91104.log.gz 2024-09-14T07:01:45.022 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91185.log 2024-09-14T07:01:45.022 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85126.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85126.log.gz 2024-09-14T07:01:45.022 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65386.log 2024-09-14T07:01:45.023 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91185.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91185.log.gz 2024-09-14T07:01:45.023 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90463.log 2024-09-14T07:01:45.023 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65386.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65386.log.gz 2024-09-14T07:01:45.026 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39767.log 2024-09-14T07:01:45.026 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90463.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90463.log.gz 2024-09-14T07:01:45.026 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73953.log 2024-09-14T07:01:45.026 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49273.log 2024-09-14T07:01:45.026 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.73953.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73953.log.gz 2024-09-14T07:01:45.026 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79815.log 2024-09-14T07:01:45.029 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.49273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39009.log 2024-09-14T07:01:45.031 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.79815.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.79815.log.gz 2024-09-14T07:01:45.032 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75129.log 2024-09-14T07:01:45.032 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85268.log 2024-09-14T07:01:45.034 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75129.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75129.log.gz 2024-09-14T07:01:45.034 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47119.log 2024-09-14T07:01:45.034 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85268.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.85268.log.gz 2024-09-14T07:01:45.035 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42478.log 2024-09-14T07:01:45.035 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40480.log 2024-09-14T07:01:45.037 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.42478.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91838.log.gz 2024-09-14T07:01:45.037 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79738.log 2024-09-14T07:01:45.038 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66658.log 2024-09-14T07:01:45.041 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92084.log 2024-09-14T07:01:45.041 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79738.log.gz 2024-09-14T07:01:45.041 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66658.log: gzip -5 --verbose -- 87.9% /var/log/ceph/cluster2-client.mirror.88915.log 2024-09-14T07:01:45.041 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66658.log.gz 2024-09-14T07:01:45.041 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87202.log 2024-09-14T07:01:45.043 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.88915.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42985.log.gz 2024-09-14T07:01:45.043 INFO:teuthology.orchestra.run.smithi170.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92084.log.gz 2024-09-14T07:01:45.043 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43299.log.gz 2024-09-14T07:01:45.044 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90730.log 2024-09-14T07:01:45.044 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87202.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87202.log.gz 2024-09-14T07:01:45.044 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44798.log 2024-09-14T07:01:45.044 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90730.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44766.log.gz 2024-09-14T07:01:45.047 INFO:teuthology.orchestra.run.smithi170.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.90730.log.gz 2024-09-14T07:01:45.047 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43949.log 2024-09-14T07:01:45.047 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.44798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91665.log 2024-09-14T07:01:45.047 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53497.log 2024-09-14T07:01:45.048 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91665.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91665.log.gz 2024-09-14T07:01:45.050 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42283.log 2024-09-14T07:01:45.050 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.53497.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53497.log.gz 2024-09-14T07:01:45.050 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41271.log 2024-09-14T07:01:45.054 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42283.log: /var/log/ceph/cluster1-client.mirror.41271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41194.log 2024-09-14T07:01:45.054 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43489.log 2024-09-14T07:01:45.054 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79154.log 2024-09-14T07:01:45.055 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37997.log 2024-09-14T07:01:45.057 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.79154.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.79154.log.gz 2024-09-14T07:01:45.061 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90636.log 2024-09-14T07:01:45.061 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37997.log.gz 2024-09-14T07:01:45.061 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68192.log.gz 2024-09-14T07:01:45.061 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72377.log 2024-09-14T07:01:45.061 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78052.log 2024-09-14T07:01:45.063 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.72377.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.72377.log.gz 2024-09-14T07:01:45.063 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49462.log 2024-09-14T07:01:45.064 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.78052.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78052.log.gz 2024-09-14T07:01:45.065 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52442.log 2024-09-14T07:01:45.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.49462.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49462.log.gz 2024-09-14T07:01:45.066 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74743.log 2024-09-14T07:01:45.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.52442.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47119.log.gz 2024-09-14T07:01:45.066 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52442.log.gz 2024-09-14T07:01:45.069 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65772.log 2024-09-14T07:01:45.069 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.74743.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74743.log.gz 2024-09-14T07:01:45.070 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45049.log 2024-09-14T07:01:45.070 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39825.log.gz 2024-09-14T07:01:45.070 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65772.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39767.log.gz 2024-09-14T07:01:45.070 INFO:teuthology.orchestra.run.smithi170.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65772.log.gz 2024-09-14T07:01:45.070 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44245.log 2024-09-14T07:01:45.072 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45049.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40480.log.gz 2024-09-14T07:01:45.072 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87269.log 2024-09-14T07:01:45.072 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.44245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66375.log 2024-09-14T07:01:45.072 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87269.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87269.log.gz 2024-09-14T07:01:45.072 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39302.log 2024-09-14T07:01:45.074 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66375.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66375.log.gz 2024-09-14T07:01:45.074 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85885.log 2024-09-14T07:01:45.075 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43768.log 2024-09-14T07:01:45.077 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85885.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85885.log.gz 2024-09-14T07:01:45.079 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46867.log 2024-09-14T07:01:45.080 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43768.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42283.log.gz 2024-09-14T07:01:45.080 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36656.log 2024-09-14T07:01:45.080 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46867.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39302.log.gz 2024-09-14T07:01:45.081 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88238.log 2024-09-14T07:01:45.081 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36656.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36656.log.gz 2024-09-14T07:01:45.082 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-14T07:01:45.083 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88238.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88238.log.gz 2024-09-14T07:01:45.084 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70711.log 2024-09-14T07:01:45.084 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36888.log 2024-09-14T07:01:45.086 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.70711.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70711.log.gz 2024-09-14T07:01:45.086 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77163.log 2024-09-14T07:01:45.086 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36888.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36888.log.gz 2024-09-14T07:01:45.088 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87044.log 2024-09-14T07:01:45.088 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77163.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77163.log.gz 2024-09-14T07:01:45.088 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88598.log 2024-09-14T07:01:45.089 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87044.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87044.log.gz 2024-09-14T07:01:45.089 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71861.log 2024-09-14T07:01:45.090 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88598.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88598.log.gz 2024-09-14T07:01:45.090 INFO:teuthology.orchestra.run.smithi170.stderr: 89.5% 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46195.log.gz 2024-09-14T07:01:45.092 INFO:teuthology.orchestra.run.smithi170.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41194.log.gz 2024-09-14T07:01:45.092 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46241.log.gz 2024-09-14T07:01:45.092 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46216.log.gz 2024-09-14T07:01:45.092 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54177.log 2024-09-14T07:01:45.092 INFO:teuthology.orchestra.run.smithi170.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44991.log.gz 2024-09-14T07:01:45.092 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45084.log.gz 2024-09-14T07:01:45.095 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.71861.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71861.log.gz 2024-09-14T07:01:45.095 INFO:teuthology.orchestra.run.smithi170.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48963.log.gz 2024-09-14T07:01:45.095 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80950.log 2024-09-14T07:01:45.095 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.54177.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54177.log.gz 2024-09-14T07:01:45.095 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41922.log 2024-09-14T07:01:45.097 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80950.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80950.log.gz 2024-09-14T07:01:45.097 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66935.log 2024-09-14T07:01:45.097 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42748.log 2024-09-14T07:01:45.099 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.66935.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66935.log.gz 2024-09-14T07:01:45.100 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38477.log 2024-09-14T07:01:45.100 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87427.log 2024-09-14T07:01:45.104 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.1.38477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85075.log 2024-09-14T07:01:45.105 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87427.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87427.log.gz 2024-09-14T07:01:45.105 INFO:teuthology.orchestra.run.smithi170.stderr: 84.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88485.log 2024-09-14T07:01:45.105 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46710.log.gz 2024-09-14T07:01:45.105 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48710.log.gz 2024-09-14T07:01:45.108 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85075.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85075.log.gz 2024-09-14T07:01:45.108 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51596.log 2024-09-14T07:01:45.109 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88485.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88485.log.gz 2024-09-14T07:01:45.109 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45841.log 2024-09-14T07:01:45.116 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.51596.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45994.log.gz 2024-09-14T07:01:45.127 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45352.log.gz 2024-09-14T07:01:45.127 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91359.log 2024-09-14T07:01:45.129 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45618.log.gz 2024-09-14T07:01:45.137 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76120.log 2024-09-14T07:01:45.138 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91359.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91359.log.gz 2024-09-14T07:01:45.145 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47030.log 2024-09-14T07:01:45.153 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87788.log 2024-09-14T07:01:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46619.log 2024-09-14T07:01:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87788.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87788.log.gz 2024-09-14T07:01:45.167 INFO:teuthology.orchestra.run.smithi170.stderr: 18.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38477.log.gz 2024-09-14T07:01:45.168 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42387.log.gz 2024-09-14T07:01:45.170 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.admin.42478.log.gz 2024-09-14T07:01:45.170 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56072.log 2024-09-14T07:01:45.171 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.46619.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41922.log.gz 2024-09-14T07:01:45.172 INFO:teuthology.orchestra.run.smithi170.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47030.log.gz 2024-09-14T07:01:45.175 INFO:teuthology.orchestra.run.smithi170.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43489.log.gz 2024-09-14T07:01:45.175 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43768.log.gz 2024-09-14T07:01:45.175 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44798.log.gz 2024-09-14T07:01:45.176 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45049.log.gz 2024-09-14T07:01:45.177 INFO:teuthology.orchestra.run.smithi170.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46867.log.gz 2024-09-14T07:01:45.181 INFO:teuthology.orchestra.run.smithi170.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52465.log.gz 2024-09-14T07:01:45.182 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46544.log 2024-09-14T07:01:45.183 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.56072.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56072.log.gz 2024-09-14T07:01:45.183 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38916.log 2024-09-14T07:01:45.184 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68778.log 2024-09-14T07:01:45.185 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69114.log 2024-09-14T07:01:45.202 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.68778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45931.log 2024-09-14T07:01:45.204 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69114.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69114.log.gz 2024-09-14T07:01:45.217 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67679.log 2024-09-14T07:01:45.228 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44823.log.gz 2024-09-14T07:01:45.229 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45931.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50685.log.gz 2024-09-14T07:01:45.229 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86402.log 2024-09-14T07:01:45.232 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67679.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67679.log.gz 2024-09-14T07:01:45.232 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89228.log 2024-09-14T07:01:45.235 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86402.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86402.log.gz 2024-09-14T07:01:45.235 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92308.log 2024-09-14T07:01:45.237 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89228.log: 83.2% -- replaced with /var/log/ceph/cluster2-client.mirror.89228.log.gz 2024-09-14T07:01:45.237 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85246.log 2024-09-14T07:01:45.238 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92308.log.gz 2024-09-14T07:01:45.238 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90678.log 2024-09-14T07:01:45.239 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85246.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85246.log.gz 2024-09-14T07:01:45.239 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41979.log 2024-09-14T07:01:45.240 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90678.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90678.log.gz 2024-09-14T07:01:45.241 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91434.log 2024-09-14T07:01:45.243 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41979.log: -- replaced with /var/log/ceph/cluster2-client.mirror.62838.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86954.log 2024-09-14T07:01:45.243 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:45.243 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91434.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91434.log.gz 2024-09-14T07:01:45.243 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83923.log 2024-09-14T07:01:45.243 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86954.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86954.log.gz 2024-09-14T07:01:45.244 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51876.log 2024-09-14T07:01:45.245 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83923.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83923.log.gz 2024-09-14T07:01:45.248 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76476.log 2024-09-14T07:01:45.251 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.51876.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46041.log.gz 2024-09-14T07:01:45.252 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41251.log.gz 2024-09-14T07:01:45.255 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44245.log.gz 2024-09-14T07:01:45.256 INFO:teuthology.orchestra.run.smithi170.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47422.log.gz 2024-09-14T07:01:45.264 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45462.log.gz 2024-09-14T07:01:45.268 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84256.log 2024-09-14T07:01:45.275 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.76476.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42068.log.gz 2024-09-14T07:01:45.277 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.76476.log.gz 2024-09-14T07:01:45.283 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46619.log.gz 2024-09-14T07:01:45.285 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46544.log.gz 2024-09-14T07:01:45.299 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42248.log 2024-09-14T07:01:45.301 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84256.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84256.log.gz 2024-09-14T07:01:45.314 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88328.log 2024-09-14T07:01:45.319 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42248.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76120.log.gz 2024-09-14T07:01:45.320 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92815.log 2024-09-14T07:01:45.321 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88328.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88328.log.gz 2024-09-14T07:01:45.322 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45161.log 2024-09-14T07:01:45.323 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92815.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92815.log.gz 2024-09-14T07:01:45.323 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44561.log 2024-09-14T07:01:45.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38660.log 2024-09-14T07:01:45.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69475.log 2024-09-14T07:01:45.325 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.38660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73793.log 2024-09-14T07:01:45.327 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.69475.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69475.log.gz 2024-09-14T07:01:45.327 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83952.log 2024-09-14T07:01:45.327 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73793.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73793.log.gz 2024-09-14T07:01:45.328 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61959.log 2024-09-14T07:01:45.329 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83952.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83952.log.gz 2024-09-14T07:01:45.329 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88665.log 2024-09-14T07:01:45.331 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.61959.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61959.log.gz 2024-09-14T07:01:45.337 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86311.log 2024-09-14T07:01:45.341 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88665.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88665.log.gz 2024-09-14T07:01:45.344 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60833.log 2024-09-14T07:01:45.345 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86311.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86311.log.gz 2024-09-14T07:01:45.345 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63946.log 2024-09-14T07:01:45.347 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45974.log.gz 2024-09-14T07:01:45.349 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.60833.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60833.log.gz 2024-09-14T07:01:45.352 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73121.log 2024-09-14T07:01:45.360 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63946.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63946.log.gz 2024-09-14T07:01:45.360 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38950.log 2024-09-14T07:01:45.362 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73121.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42248.log.gz 2024-09-14T07:01:45.364 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73121.log.gz 2024-09-14T07:01:45.364 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47574.log 2024-09-14T07:01:45.365 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.38950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85914.log 2024-09-14T07:01:45.365 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87292.log 2024-09-14T07:01:45.366 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85914.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40572.log.gz 2024-09-14T07:01:45.367 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85914.log.gz 2024-09-14T07:01:45.367 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92605.log 2024-09-14T07:01:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87292.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87292.log.gz 2024-09-14T07:01:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67699.log 2024-09-14T07:01:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92605.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92605.log.gz 2024-09-14T07:01:45.370 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85754.log 2024-09-14T07:01:45.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.67699.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67699.log.gz 2024-09-14T07:01:45.377 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89586.log 2024-09-14T07:01:45.377 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85754.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85754.log.gz 2024-09-14T07:01:45.378 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40592.log 2024-09-14T07:01:45.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89586.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89586.log.gz 2024-09-14T07:01:45.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63735.log 2024-09-14T07:01:45.382 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.40592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48515.log 2024-09-14T07:01:45.383 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.63735.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55661.log.gz 2024-09-14T07:01:45.384 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71208.log 2024-09-14T07:01:45.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48515.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63735.log.gz 2024-09-14T07:01:45.384 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77514.log 2024-09-14T07:01:45.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77514.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77514.log.gz 2024-09-14T07:01:45.387 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84427.log 2024-09-14T07:01:45.388 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40015.log 2024-09-14T07:01:45.388 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84427.log: /var/log/ceph/cluster2-client.mirror.71208.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71208.log.gz 2024-09-14T07:01:45.389 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84427.log.gz 2024-09-14T07:01:45.389 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49174.log 2024-09-14T07:01:45.390 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77626.log 2024-09-14T07:01:45.402 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.49174.log: 95.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79770.log 2024-09-14T07:01:45.402 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51596.log.gz 2024-09-14T07:01:45.402 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.77626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61504.log 2024-09-14T07:01:45.402 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.79770.log: -- /var/log/ceph/cluster2-client.mirror.40641.log 2024-09-14T07:01:45.402 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79770.log.gz/var/log/ceph/cluster2-client.mirror.61504.log: 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68778.log.gz 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79837.log 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.40641.log: -- replaced with /var/log/ceph/cluster2-client.mirror.61504.log.gz 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58348.log 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86431.log 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.58348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84307.log 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58348.log.gz 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39360.log 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84307.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84307.log.gz 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.86431.log.gz 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77626.log.gz 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.91125.log 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75263.log 2024-09-14T07:01:45.403 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45419.log 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.75263.log: -- replaced with /var/log/ceph/cluster2-client.mirror.91125.log.gz 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81042.log 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76053.log 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81042.log: /var/log/ceph/cluster1-client.mirror.45419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89546.log 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.76053.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76053.log.gz 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81042.log.gz 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44843.log 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62704.log 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.44843.log: /var/log/ceph/cluster2-client.mirror.89546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56095.log 2024-09-14T07:01:45.404 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62704.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62704.log.gz 2024-09-14T07:01:45.414 INFO:teuthology.orchestra.run.smithi170.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89546.log.gz 2024-09-14T07:01:45.414 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38472.log 2024-09-14T07:01:45.414 INFO:teuthology.orchestra.run.smithi170.stderr: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79837.log.gz 2024-09-14T07:01:45.414 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.56095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69520.log 2024-09-14T07:01:45.414 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.38472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37751.log 2024-09-14T07:01:45.414 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90825.log 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37751.log: /var/log/ceph/cluster1-client.mirror.69520.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37751.log.gz 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62325.log 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90825.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90825.log.gz 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88819.log 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62325.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69520.log.gz 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85843.log 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.88819.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62325.log.gz 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr: 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88819.log.gz 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38261.log 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69714.log 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.38261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87765.log 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69714.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69714.log.gz 2024-09-14T07:01:45.415 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85843.log.gz 2024-09-14T07:01:45.416 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75077.log 2024-09-14T07:01:45.416 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87765.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.87765.log.gz 2024-09-14T07:01:45.416 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80467.log 2024-09-14T07:01:45.418 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75077.log: 94.1% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-14T07:01:45.418 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48337.log 2024-09-14T07:01:45.418 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80467.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80467.log.gz 2024-09-14T07:01:45.419 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45595.log 2024-09-14T07:01:45.419 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86260.log 2024-09-14T07:01:45.421 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72718.log 2024-09-14T07:01:45.424 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86260.log: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75077.log.gz 2024-09-14T07:01:45.424 INFO:teuthology.orchestra.run.smithi170.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86260.log.gz 2024-09-14T07:01:45.427 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90127.log 2024-09-14T07:01:45.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.72718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57289.log 2024-09-14T07:01:45.427 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90127.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72718.log.gz 2024-09-14T07:01:45.427 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89484.log 2024-09-14T07:01:45.430 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.57289.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90127.log.gz 2024-09-14T07:01:45.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60890.log 2024-09-14T07:01:45.432 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89484.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89484.log.gz 2024-09-14T07:01:45.432 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85104.log 2024-09-14T07:01:45.433 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.60890.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60890.log.gz 2024-09-14T07:01:45.436 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42340.log 2024-09-14T07:01:45.437 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85104.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85104.log.gz 2024-09-14T07:01:45.439 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44270.log 2024-09-14T07:01:45.439 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84762.log 2024-09-14T07:01:45.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75055.log 2024-09-14T07:01:45.441 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84762.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84762.log.gz 2024-09-14T07:01:45.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91516.log 2024-09-14T07:01:45.443 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.75055.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75055.log.gz 2024-09-14T07:01:45.445 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69134.log 2024-09-14T07:01:45.445 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91516.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.91516.log.gz 2024-09-14T07:01:45.446 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60932.log 2024-09-14T07:01:45.450 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.69134.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69134.log.gz 2024-09-14T07:01:45.450 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88915.log.gz 2024-09-14T07:01:45.450 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87494.log 2024-09-14T07:01:45.452 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88080.log 2024-09-14T07:01:45.452 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87494.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87494.log.gz 2024-09-14T07:01:45.452 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87855.log 2024-09-14T07:01:45.455 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88080.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88080.log.gz 2024-09-14T07:01:45.455 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77985.log 2024-09-14T07:01:45.455 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87855.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87855.log.gz 2024-09-14T07:01:45.455 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44489.log 2024-09-14T07:01:45.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77985.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77985.log.gz 2024-09-14T07:01:45.457 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79360.log 2024-09-14T07:01:45.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44956.log 2024-09-14T07:01:45.458 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79360.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79360.log.gz 2024-09-14T07:01:45.459 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.60932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71164.log 2024-09-14T07:01:45.459 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-14T07:01:45.469 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.71164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90705.log 2024-09-14T07:01:45.477 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-mon.a.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49174.log.gz 2024-09-14T07:01:45.480 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40015.log.gz 2024-09-14T07:01:45.480 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90102.log 2024-09-14T07:01:45.481 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90705.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90705.log.gz 2024-09-14T07:01:45.489 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47340.log 2024-09-14T07:01:45.489 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90102.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90102.log.gz 2024-09-14T07:01:45.492 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39360.log.gz 2024-09-14T07:01:45.493 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48515.log.gz 2024-09-14T07:01:45.496 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42748.log.gz 2024-09-14T07:01:45.502 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81062.log 2024-09-14T07:01:45.511 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47340.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89295.log.gz 2024-09-14T07:01:45.512 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85339.log 2024-09-14T07:01:45.513 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.81062.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.81062.log.gz 2024-09-14T07:01:45.515 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54531.log 2024-09-14T07:01:45.515 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85339.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85339.log.gz 2024-09-14T07:01:45.516 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43902.log 2024-09-14T07:01:45.516 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.54531.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54531.log.gz 2024-09-14T07:01:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41059.log 2024-09-14T07:01:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60910.log 2024-09-14T07:01:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.41059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43206.log 2024-09-14T07:01:45.521 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.60910.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60910.log.gz 2024-09-14T07:01:45.522 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44731.log 2024-09-14T07:01:45.522 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83479.log 2024-09-14T07:01:45.522 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91750.log 2024-09-14T07:01:45.522 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83479.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83479.log.gz 2024-09-14T07:01:45.525 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91494.log 2024-09-14T07:01:45.531 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91750.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91750.log.gz 2024-09-14T07:01:45.531 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65981.log 2024-09-14T07:01:45.538 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91494.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91494.log.gz 2024-09-14T07:01:45.539 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90610.log 2024-09-14T07:01:45.543 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.65981.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65981.log.gz 2024-09-14T07:01:45.547 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82123.log 2024-09-14T07:01:45.547 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90610.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90610.log.gz 2024-09-14T07:01:45.547 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89112.log 2024-09-14T07:01:45.550 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82123.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51876.log.gz 2024-09-14T07:01:45.551 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63182.log 2024-09-14T07:01:45.551 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89112.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89112.log.gz 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36889.log 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63182.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82123.log.gz 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63889.log 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36889.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36889.log.gz 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42026.log 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63182.log.gz 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63889.log: gzip -5 --verbose -- 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63889.log.gz 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.46480.log 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48538.log 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38582.log 2024-09-14T07:01:45.556 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64499.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.1.38582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90157.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.64499.log: -- /var/log/ceph/cluster2-client.mirror.49628.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64499.log.gz/var/log/ceph/cluster1-client.mirror.90157.log: 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90157.log.gz 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43582.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.49628.log: gzip -5 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49628.log.gz 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.84833.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88802.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84833.log: /var/log/ceph/cluster2-client.mirror.43582.log: gzip -5 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84833.log.gz 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.61914.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45399.log 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.61914.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61914.log.gz 2024-09-14T07:01:45.566 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88802.log: 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.88802.log.gz 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41159.log 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72332.log 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.41159.log: --verbose -- /var/log/ceph/cluster2-client.mirror.63468.log 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50416.log 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.63468.log: /var/log/ceph/cluster1-client.mirror.72332.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63468.log.gz 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72332.log.gz 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82407.log 2024-09-14T07:01:45.567 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50416.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50416.log.gz 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90636.log.gz 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92212.log 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41028.log 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92212.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92212.log.gz 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82407.log.gz 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92380.log 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.41028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39267.log 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92380.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92380.log.gz 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster1-client.mirror.70332.log 2024-09-14T07:01:45.581 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43845.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.70332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37121.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.43845.log: -- replaced with /var/log/ceph/cluster1-client.mirror.70332.log.gz 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47237.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37121.log: /var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37121.log.gz 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster1-client.mirror.90408.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38981.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90408.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90408.log.gz 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster1-client.mirror.88215.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.38981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84214.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88215.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88215.log.gz 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90263.log 2024-09-14T07:01:45.582 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44313.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90263.log: gzip 83.4% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68822.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84214.log.gz 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.90263.log.gz 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54893.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44313.log: /var/log/ceph/cluster2-client.mirror.68822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46392.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.54893.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54893.log.gz 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68822.log.gz 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41298.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39174.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90783.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86453.log/var/log/ceph/cluster2-client.mirror.39174.log: 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90783.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90783.log.gz 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39918.log 2024-09-14T07:01:45.583 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38577.log 2024-09-14T07:01:45.584 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86453.log.gz 2024-09-14T07:01:45.584 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72686.log 2024-09-14T07:01:45.584 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88889.log 2024-09-14T07:01:45.584 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38577.log: /var/log/ceph/cluster2-client.mirror.72686.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72686.log.gz 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84378.log 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.88889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70756.log 87.5% 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.88889.log.gz 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84378.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84378.log.gz 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42121.log 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71164.log.gz 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.70756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82236.log 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.42121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45266.log 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84711.log/var/log/ceph/cluster2-client.mirror.82236.log: 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.45266.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70756.log.gz 2024-09-14T07:01:45.596 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.mirror.40294.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.84711.log: -- /var/log/ceph/cluster1-client.mirror.63202.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40937.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82236.log.gz 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.63202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53520.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40937.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84711.log.gz 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87112.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.53520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39639.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56883.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39639.log: /var/log/ceph/cluster1-client.mirror.87112.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87112.log.gz 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90760.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63202.log.gz 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.56883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76498.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83386.log 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.90760.log: /var/log/ceph/cluster2-client.mirror.76498.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90760.log.gz 2024-09-14T07:01:45.597 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47947.log 2024-09-14T07:01:45.598 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83386.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83386.log.gz 2024-09-14T07:01:45.598 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41659.log 2024-09-14T07:01:45.598 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47666.log 2024-09-14T07:01:45.598 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75241.log 2024-09-14T07:01:45.598 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83195.log 2024-09-14T07:01:45.598 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.75241.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75241.log.gz 2024-09-14T07:01:45.599 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87810.log 2024-09-14T07:01:45.600 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83195.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83195.log.gz 2024-09-14T07:01:45.600 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42091.log 2024-09-14T07:01:45.601 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87810.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87810.log.gz 2024-09-14T07:01:45.601 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77927.log 2024-09-14T07:01:45.602 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64524.log 2024-09-14T07:01:45.603 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.77927.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77927.log.gz 2024-09-14T07:01:45.605 INFO:teuthology.orchestra.run.smithi170.stderr: 88.9% 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44561.log.gz 2024-09-14T07:01:45.605 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44731.log.gz 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44956.log.gz 2024-09-14T07:01:45.605 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:45.607 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85632.log 2024-09-14T07:01:45.611 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.64524.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39639.log.gz 2024-09-14T07:01:45.613 INFO:teuthology.orchestra.run.smithi170.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64524.log.gz 2024-09-14T07:01:45.614 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41679.log 2024-09-14T07:01:45.615 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39174.log.gz 2024-09-14T07:01:45.615 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39267.log.gz 2024-09-14T07:01:45.618 INFO:teuthology.orchestra.run.smithi170.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41159.log.gz 2024-09-14T07:01:45.623 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45931.log.gz 2024-09-14T07:01:45.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85632.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85632.log.gz 2024-09-14T07:01:45.626 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84811.log 2024-09-14T07:01:45.626 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80201.log 2024-09-14T07:01:45.626 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84811.log: /var/log/ceph/cluster1-client.mirror.41679.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41271.log.gz 2024-09-14T07:01:45.627 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84811.log.gz 2024-09-14T07:01:45.628 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51301.log 2024-09-14T07:01:45.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.80201.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80201.log.gz 2024-09-14T07:01:45.630 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62300.log 2024-09-14T07:01:45.631 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.51301.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51301.log.gz 2024-09-14T07:01:45.631 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40108.log 2024-09-14T07:01:45.634 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.62300.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62300.log.gz 2024-09-14T07:01:45.635 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58129.log 2024-09-14T07:01:45.639 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68150.log 2024-09-14T07:01:45.642 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.58129.log: 88.5% 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.41059.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.41028.log.gz 2024-09-14T07:01:45.642 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:45.646 INFO:teuthology.orchestra.run.smithi170.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57289.log.gz 2024-09-14T07:01:45.646 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87878.log 2024-09-14T07:01:45.648 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.68150.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68150.log.gz 2024-09-14T07:01:45.650 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89944.log 2024-09-14T07:01:45.653 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87878.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45419.log.gz 2024-09-14T07:01:45.653 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45595.log.gz 2024-09-14T07:01:45.655 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87878.log.gz 2024-09-14T07:01:45.655 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63157.log 2024-09-14T07:01:45.655 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89944.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47947.log.gz 2024-09-14T07:01:45.656 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48337.log.gz 2024-09-14T07:01:45.656 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89944.log.gz 2024-09-14T07:01:45.659 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1%gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.63157.log: -- /var/log/ceph/cluster2-client.mirror.40890.log 2024-09-14T07:01:45.659 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42340.log.gz 2024-09-14T07:01:45.659 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63157.log.gz 2024-09-14T07:01:45.659 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66890.log 2024-09-14T07:01:45.659 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87562.log 2024-09-14T07:01:45.660 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.66890.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66890.log.gz 2024-09-14T07:01:45.660 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37373.log 2024-09-14T07:01:45.664 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87562.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87562.log.gz 2024-09-14T07:01:45.664 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44866.log 2024-09-14T07:01:45.665 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37373.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37373.log.gz 2024-09-14T07:01:45.667 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46413.log 2024-09-14T07:01:45.667 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74606.log 2024-09-14T07:01:45.667 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43922.log 2024-09-14T07:01:45.676 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.74606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86007.log 2024-09-14T07:01:45.681 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.43922.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75263.log.gz 2024-09-14T07:01:45.683 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85490.log 2024-09-14T07:01:45.684 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86007.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86007.log.gz 2024-09-14T07:01:45.686 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76750.log 2024-09-14T07:01:45.686 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85490.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38660.log.gz 2024-09-14T07:01:45.687 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47237.log.gz 2024-09-14T07:01:45.687 INFO:teuthology.orchestra.run.smithi170.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47574.log.gz 2024-09-14T07:01:45.688 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47666.log.gz 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47340.log.gz 2024-09-14T07:01:45.688 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:45.688 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85490.log.gz 2024-09-14T07:01:45.690 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45775.log 2024-09-14T07:01:45.690 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76750.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76750.log.gz 2024-09-14T07:01:45.692 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82680.log 2024-09-14T07:01:45.694 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45775.log: 88.2% 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.38950.log.gz 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.38981.log.gz 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88170.log 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43206.log.gz 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82680.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45841.log.gz 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43582.log.gz 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82680.log.gz 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91001.log 2024-09-14T07:01:45.695 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43845.log.gz 2024-09-14T07:01:45.696 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88170.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46480.log.gz 2024-09-14T07:01:45.696 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40294.log.gz 2024-09-14T07:01:45.697 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39918.log.gz 2024-09-14T07:01:45.697 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40108.log.gz 2024-09-14T07:01:45.704 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88170.log.gz 2024-09-14T07:01:45.705 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45509.log 2024-09-14T07:01:45.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91001.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91001.log.gz 2024-09-14T07:01:45.708 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37583.log 2024-09-14T07:01:45.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38577.log 2024-09-14T07:01:45.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37583.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37583.log.gz 2024-09-14T07:01:45.710 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80985.log 2024-09-14T07:01:45.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.0.38577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44050.log 2024-09-14T07:01:45.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80985.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80985.log.gz 2024-09-14T07:01:45.711 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44213.log 2024-09-14T07:01:45.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.44050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89825.log 2024-09-14T07:01:45.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61376.log 2024-09-14T07:01:45.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89825.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.89825.log.gz 2024-09-14T07:01:45.716 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43969.log 2024-09-14T07:01:45.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61376.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61376.log.gz 2024-09-14T07:01:45.718 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69156.log 2024-09-14T07:01:45.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.43969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66526.log 2024-09-14T07:01:45.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88620.log 2024-09-14T07:01:45.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66526.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66526.log.gz 2024-09-14T07:01:45.738 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38055.log 2024-09-14T07:01:45.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88620.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88620.log.gz 2024-09-14T07:01:45.746 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53520.log.gz 2024-09-14T07:01:45.747 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47142.log 2024-09-14T07:01:45.748 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.38055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91610.log 2024-09-14T07:01:45.749 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46929.log 2024-09-14T07:01:45.749 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91610.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.91610.log.gz 2024-09-14T07:01:45.751 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48167.log 2024-09-14T07:01:45.752 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86218.log 2024-09-14T07:01:45.752 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87743.log 2024-09-14T07:01:45.753 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86218.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86218.log.gz 2024-09-14T07:01:45.755 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89984.log 2024-09-14T07:01:45.755 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87743.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87743.log.gz 2024-09-14T07:01:45.757 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85732.log 2024-09-14T07:01:45.757 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89984.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89984.log.gz 2024-09-14T07:01:45.757 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69542.log 2024-09-14T07:01:45.757 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85732.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85732.log.gz 2024-09-14T07:01:45.758 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68058.log 2024-09-14T07:01:45.760 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69542.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40592.log.gz 2024-09-14T07:01:45.765 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74606.log.gz 2024-09-14T07:01:45.766 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43640.log 2024-09-14T07:01:45.786 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.68058.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68058.log.gz 2024-09-14T07:01:45.786 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88035.log 2024-09-14T07:01:45.786 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46392.log.gz 2024-09-14T07:01:45.786 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46413.log.gz 2024-09-14T07:01:45.786 INFO:teuthology.orchestra.run.smithi170.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69542.log.gz 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85470.log 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45223.log 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88035.log.gz 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93082.log 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70100.log 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85470.log.gz 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.93082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84620.log 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93082.log.gz 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82307.log 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84620.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70100.log.gz 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84620.log.gz 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32591.log 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6%/var/log/ceph/cluster2-client.mirror.82307.log: -- replaced with /var/log/ceph/cluster2-client.mirror.69156.log.gz 2024-09-14T07:01:45.787 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92237.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.32591.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32591.log.gz 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71629.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82307.log.gz 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.92237.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92237.log.gz 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89924.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56095.log.gz 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.71629.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71629.log.gz 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92356.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/cluster2-client.mirror.89924.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82842.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.92356.log: 88.3% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92356.log.gz 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -- replaced with /var/log/ceph/cluster2-client.mirror.89924.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70013.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46283.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82842.log: /var/log/ceph/cluster2-client.mirror.70013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54554.log 2024-09-14T07:01:45.788 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84640.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42360.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84640.log: /var/log/ceph/cluster2-client.mirror.54554.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82842.log.gz 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45642.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.42360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38021.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7%/var/log/ceph/cluster1-client.mirror.45642.log: -- replaced with /var/log/ceph/cluster2-client.mirror.84640.log.gz 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82336.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.38021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62771.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74650.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.62771.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82336.log.gz 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.88836.log 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45161.log.gz 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62771.log.gz 2024-09-14T07:01:45.789 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.74650.log: gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74650.log.gz 2024-09-14T07:01:45.790 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.39116.log 2024-09-14T07:01:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80223.log 2024-09-14T07:01:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.39116.log: /var/log/ceph/cluster1-client.mirror.0.88836.log: 18.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.88836.log.gz 2024-09-14T07:01:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91272.log 2024-09-14T07:01:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42452.log 2024-09-14T07:01:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91272.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91272.log.gz 2024-09-14T07:01:45.790 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43171.log 2024-09-14T07:01:45.792 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.42452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76456.log 2024-09-14T07:01:45.792 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50939.log 2024-09-14T07:01:45.792 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76456.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76456.log.gz 2024-09-14T07:01:45.794 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91688.log 2024-09-14T07:01:45.796 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.50939.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76498.log.gz 2024-09-14T07:01:45.796 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42026.log.gz 2024-09-14T07:01:45.798 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37499.log 2024-09-14T07:01:45.799 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91688.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91688.log.gz 2024-09-14T07:01:45.801 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40329.log 2024-09-14T07:01:45.801 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.37499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37499.log.gz 2024-09-14T07:01:45.801 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73064.log 2024-09-14T07:01:45.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89166.log 2024-09-14T07:01:45.810 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73064.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73064.log.gz 2024-09-14T07:01:45.815 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89025.log 2024-09-14T07:01:45.818 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89166.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89166.log.gz 2024-09-14T07:01:45.818 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75635.log 2024-09-14T07:01:45.819 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89025.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89025.log.gz 2024-09-14T07:01:45.819 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84740.log 2024-09-14T07:01:45.820 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75635.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75635.log.gz 2024-09-14T07:01:45.820 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46174.log 2024-09-14T07:01:45.822 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84740.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84740.log.gz 2024-09-14T07:01:45.822 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38463.log 2024-09-14T07:01:45.823 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46174.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44843.log.gz 2024-09-14T07:01:45.824 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57266.log 2024-09-14T07:01:45.825 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.38463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40798.log 2024-09-14T07:01:45.825 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.57266.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57266.log.gz 2024-09-14T07:01:45.826 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84669.log 2024-09-14T07:01:45.827 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49651.log 2024-09-14T07:01:45.828 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84669.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84669.log.gz 2024-09-14T07:01:45.830 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46304.log 2024-09-14T07:01:45.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.49651.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60932.log.gz 2024-09-14T07:01:45.832 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70479.log 2024-09-14T07:01:45.832 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91730.log 2024-09-14T07:01:45.834 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70479.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70479.log.gz 2024-09-14T07:01:45.834 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46522.log 2024-09-14T07:01:45.834 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91730.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58129.log.gz 2024-09-14T07:01:45.834 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91730.log.gz 2024-09-14T07:01:45.836 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89567.log 2024-09-14T07:01:45.836 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84285.log 2024-09-14T07:01:45.836 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89567.log: /var/log/ceph/cluster2-client.mirror.46522.log: 83.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53118.log 2024-09-14T07:01:45.839 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84285.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84285.log.gz 2024-09-14T07:01:45.839 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.89567.log.gz 2024-09-14T07:01:45.839 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81806.log 2024-09-14T07:01:45.839 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.53118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88440.log 2024-09-14T07:01:45.839 INFO:teuthology.orchestra.run.smithi170.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53118.log.gz 2024-09-14T07:01:45.839 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89505.log 2024-09-14T07:01:45.839 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88440.log: /var/log/ceph/cluster2-client.mirror.81806.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88440.log.gz 2024-09-14T07:01:45.840 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81806.log.gz 2024-09-14T07:01:45.844 INFO:teuthology.orchestra.run.smithi170.stderr:gzip/var/log/ceph/cluster2-client.mirror.89505.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71122.log 2024-09-14T07:01:45.844 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.89505.log.gz 2024-09-14T07:01:45.844 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47260.log 2024-09-14T07:01:45.844 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86473.log 2024-09-14T07:01:45.844 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47260.log: /var/log/ceph/cluster2-client.mirror.71122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45376.log 2024-09-14T07:01:45.845 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86473.log: 69.5% -- replaced with /var/log/ceph/cluster1-client.admin.38055.log.gz 2024-09-14T07:01:45.845 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71122.log.gz 2024-09-14T07:01:45.845 INFO:teuthology.orchestra.run.smithi170.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.38261.log.gz 2024-09-14T07:01:45.846 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86473.log.gz 2024-09-14T07:01:45.848 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86507.log 2024-09-14T07:01:45.848 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45376.log: gzip 18.7% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89965.log 2024-09-14T07:01:45.848 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.38582.log.gz 2024-09-14T07:01:45.849 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86507.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86507.log.gz 2024-09-14T07:01:45.849 INFO:teuthology.orchestra.run.smithi170.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.admin.44050.log.gz 2024-09-14T07:01:45.850 INFO:teuthology.orchestra.run.smithi170.stderr: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46929.log.gz 2024-09-14T07:01:45.850 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43640.log.gz 2024-09-14T07:01:45.851 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40329.log.gz 2024-09-14T07:01:45.852 INFO:teuthology.orchestra.run.smithi170.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46174.log.gz 2024-09-14T07:01:45.854 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79795.log 2024-09-14T07:01:45.854 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.89965.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89965.log.gz 2024-09-14T07:01:45.855 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65027.log 2024-09-14T07:01:45.858 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79795.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79795.log.gz 2024-09-14T07:01:45.858 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41002.log 2024-09-14T07:01:45.860 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.65027.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65027.log.gz 2024-09-14T07:01:45.860 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87224.log 2024-09-14T07:01:45.862 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.41002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65049.log 2024-09-14T07:01:45.862 INFO:teuthology.orchestra.run.smithi170.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39116.log.gz 2024-09-14T07:01:45.862 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87224.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43949.log.gz 2024-09-14T07:01:45.862 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87224.log.gz 2024-09-14T07:01:45.863 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48190.log 2024-09-14T07:01:45.882 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40201.log 2024-09-14T07:01:45.893 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73828.log 2024-09-14T07:01:45.904 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41636.log 2024-09-14T07:01:45.904 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73828.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73828.log.gz 2024-09-14T07:01:45.912 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79176.log 2024-09-14T07:01:45.921 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.41636.log: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38639.log.gz 2024-09-14T07:01:45.924 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46262.log 2024-09-14T07:01:45.930 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79176.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.79176.log.gz 2024-09-14T07:01:45.930 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90885.log 2024-09-14T07:01:45.933 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44596.log 2024-09-14T07:01:45.933 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90885.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90885.log.gz 2024-09-14T07:01:45.933 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83315.log 2024-09-14T07:01:45.933 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65936.log 2024-09-14T07:01:45.937 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83315.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83315.log.gz 2024-09-14T07:01:45.937 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82893.log 2024-09-14T07:01:45.937 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.65936.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65936.log.gz 2024-09-14T07:01:45.937 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90933.log 2024-09-14T07:01:45.937 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82893.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82893.log.gz 2024-09-14T07:01:45.937 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54200.log 2024-09-14T07:01:45.946 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90933.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41679.log.gz 2024-09-14T07:01:45.962 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89406.log 2024-09-14T07:01:45.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.54200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76862.log 2024-09-14T07:01:45.973 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56883.log.gz 2024-09-14T07:01:45.973 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89406.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89406.log.gz 2024-09-14T07:01:45.983 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83668.log 2024-09-14T07:01:45.992 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.76862.log: 91.8% 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44313.log.gz 2024-09-14T07:01:45.992 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.76862.log.gz 2024-09-14T07:01:45.994 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80601.log 2024-09-14T07:01:45.996 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83668.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83668.log.gz 2024-09-14T07:01:46.001 INFO:teuthology.orchestra.run.smithi170.stderr: 18.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38463.log.gz 2024-09-14T07:01:46.004 INFO:teuthology.orchestra.run.smithi170.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48167.log.gz 2024-09-14T07:01:46.005 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44213.log.gz 2024-09-14T07:01:46.006 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92773.log 2024-09-14T07:01:46.008 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.80601.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43171.log.gz 2024-09-14T07:01:46.008 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44596.log.gz 2024-09-14T07:01:46.010 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46522.log.gz 2024-09-14T07:01:46.012 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40201.log.gz 2024-09-14T07:01:46.013 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40937.log.gz 2024-09-14T07:01:46.020 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41979.log.gz 2024-09-14T07:01:46.029 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86639.log 2024-09-14T07:01:46.030 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92773.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92773.log.gz 2024-09-14T07:01:46.040 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61603.log 2024-09-14T07:01:46.040 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86639.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86639.log.gz 2024-09-14T07:01:46.060 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40979.log 2024-09-14T07:01:46.080 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61603.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45755.log 2024-09-14T07:01:46.088 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40979.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44270.log.gz 2024-09-14T07:01:46.092 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60481.log 2024-09-14T07:01:46.109 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88058.log 2024-09-14T07:01:46.109 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.60481.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60481.log.gz 2024-09-14T07:01:46.111 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80223.log.gz 2024-09-14T07:01:46.113 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68479.log 2024-09-14T07:01:46.113 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88058.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88058.log.gz 2024-09-14T07:01:46.114 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41393.log 2024-09-14T07:01:46.118 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.68479.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68479.log.gz 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41298.log.gz 2024-09-14T07:01:46.118 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:01:46.118 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47363.log 2024-09-14T07:01:46.118 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.41393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43078.log 2024-09-14T07:01:46.118 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.47363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85217.log 2024-09-14T07:01:46.118 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72207.log 2024-09-14T07:01:46.120 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85217.log: 95.6% 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85217.log.gz 2024-09-14T07:01:46.120 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78020.log 2024-09-14T07:01:46.120 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.72207.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72207.log.gz 2024-09-14T07:01:46.122 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82800.log 2024-09-14T07:01:46.122 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78020.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78020.log.gz 2024-09-14T07:01:46.123 INFO:teuthology.orchestra.run.smithi170.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45266.log.gz 2024-09-14T07:01:46.124 INFO:teuthology.orchestra.run.smithi170.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45223.log.gz 2024-09-14T07:01:46.124 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85703.log 2024-09-14T07:01:46.126 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82800.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82800.log.gz 2024-09-14T07:01:46.126 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43733.log 2024-09-14T07:01:46.130 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85703.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85703.log.gz 2024-09-14T07:01:46.130 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76431.log 2024-09-14T07:01:46.131 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73549.log 2024-09-14T07:01:46.131 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.76431.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76431.log.gz 2024-09-14T07:01:46.131 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82429.log 2024-09-14T07:01:46.131 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.73549.log: -- replaced with /var/log/ceph/cluster2-client.mirror.80601.log.gz 2024-09-14T07:01:46.139 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41659.log.gz 2024-09-14T07:01:46.150 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81826.log 2024-09-14T07:01:46.152 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.82429.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82429.log.gz 2024-09-14T07:01:46.171 INFO:teuthology.orchestra.run.smithi170.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65049.log.gz 2024-09-14T07:01:46.171 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87157.log 2024-09-14T07:01:46.172 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.81826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87968.log 2024-09-14T07:01:46.172 INFO:teuthology.orchestra.run.smithi170.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.81826.log.gz 2024-09-14T07:01:46.174 INFO:teuthology.orchestra.run.smithi170.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.38021.log.gz 2024-09-14T07:01:46.175 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.42121.log.gz 2024-09-14T07:01:46.176 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43922.log.gz 2024-09-14T07:01:46.179 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87157.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87157.log.gz 2024-09-14T07:01:46.180 INFO:teuthology.orchestra.run.smithi170.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43078.log.gz 2024-09-14T07:01:46.182 INFO:teuthology.orchestra.run.smithi170.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43733.log.gz 2024-09-14T07:01:46.184 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38243.log 2024-09-14T07:01:46.184 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87968.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87968.log.gz 2024-09-14T07:01:46.187 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40798.log.gz 2024-09-14T07:01:46.189 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70013.log.gz 2024-09-14T07:01:46.190 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.38243.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46262.log.gz 2024-09-14T07:01:46.191 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46304.log.gz 2024-09-14T07:01:46.196 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59239.log 2024-09-14T07:01:46.196 INFO:teuthology.orchestra.run.smithi170.stderr:gzip 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46283.log.gz 2024-09-14T07:01:46.196 INFO:teuthology.orchestra.run.smithi170.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92162.log 2024-09-14T07:01:46.196 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.59239.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59239.log.gz 2024-09-14T07:01:46.196 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90433.log 2024-09-14T07:01:46.196 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.92162.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92162.log.gz 2024-09-14T07:01:46.196 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91897.log 2024-09-14T07:01:46.198 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90433.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90433.log.gz 2024-09-14T07:01:46.198 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88260.log 2024-09-14T07:01:46.201 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.91897.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91897.log.gz 2024-09-14T07:01:46.201 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78438.log 2024-09-14T07:01:46.201 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88260.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88260.log.gz 2024-09-14T07:01:46.201 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38140.log 2024-09-14T07:01:46.203 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.78438.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78438.log.gz 2024-09-14T07:01:46.203 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.41393.log.gz 2024-09-14T07:01:46.204 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90032.log 2024-09-14T07:01:46.204 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38639.log 2024-09-14T07:01:46.205 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90032.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43902.log.gz 2024-09-14T07:01:46.211 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88733.log 2024-09-14T07:01:46.221 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.1.38639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37289.log 2024-09-14T07:01:46.221 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88733.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88733.log.gz 2024-09-14T07:01:46.232 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83215.log 2024-09-14T07:01:46.233 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37289.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37289.log.gz 2024-09-14T07:01:46.239 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78406.log 2024-09-14T07:01:46.241 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83215.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83215.log.gz 2024-09-14T07:01:46.256 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38123.log 2024-09-14T07:01:46.257 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78406.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78406.log.gz 2024-09-14T07:01:46.263 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43969.log.gz 2024-09-14T07:01:46.265 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45861.log 2024-09-14T07:01:46.274 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79459.log 2024-09-14T07:01:46.274 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45861.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44866.log.gz 2024-09-14T07:01:46.275 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45642.log.gz 2024-09-14T07:01:46.278 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54554.log.gz 2024-09-14T07:01:46.279 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84882.log 2024-09-14T07:01:46.301 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.79459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83810.log 2024-09-14T07:01:46.302 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84882.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84882.log.gz 2024-09-14T07:01:46.315 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83357.log 2024-09-14T07:01:46.316 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83810.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83810.log.gz 2024-09-14T07:01:46.332 INFO:teuthology.orchestra.run.smithi170.stderr: 57.1% 63.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38123.log.gz 2024-09-14T07:01:46.332 INFO:teuthology.orchestra.run.smithi170.stderr: 80.3% 95.4% -- replaced with /var/log/ceph/cluster2-client.admin.38243.log.gz 2024-09-14T07:01:46.332 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.90933.log.gz 2024-09-14T07:01:46.332 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38140.log.gz 2024-09-14T07:01:46.335 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83357.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40890.log.gz 2024-09-14T07:01:46.335 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62796.log 2024-09-14T07:01:46.335 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83357.log.gz 2024-09-14T07:01:46.335 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-14T07:01:46.337 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.62796.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62796.log.gz 2024-09-14T07:01:46.338 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70736.log 2024-09-14T07:01:46.340 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45509.log.gz 2024-09-14T07:01:46.342 INFO:teuthology.orchestra.run.smithi170.stderr: 87.1% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-14T07:01:46.342 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36317.log 2024-09-14T07:01:46.343 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70736.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54200.log.gz 2024-09-14T07:01:46.344 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70736.log.gz 2024-09-14T07:01:46.345 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40143.log 2024-09-14T07:01:46.345 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36317.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36317.log.gz 2024-09-14T07:01:46.346 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45728.log 2024-09-14T07:01:46.346 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.40143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48360.log 2024-09-14T07:01:46.347 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85683.log 2024-09-14T07:01:46.349 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67587.log 2024-09-14T07:01:46.349 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69679.log 2024-09-14T07:01:46.349 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.67587.log: /var/log/ceph/cluster2-client.mirror.85683.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67587.log.gz 2024-09-14T07:01:46.349 INFO:teuthology.orchestra.run.smithi170.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85683.log.gz 2024-09-14T07:01:46.351 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75934.log 2024-09-14T07:01:46.352 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70399.log 2024-09-14T07:01:46.353 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75934.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81852.log 2024-09-14T07:01:46.356 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.70399.log: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69679.log.gz 2024-09-14T07:01:46.356 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.75934.log.gz 2024-09-14T07:01:46.357 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76817.log 2024-09-14T07:01:46.358 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81852.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81852.log.gz 2024-09-14T07:01:46.359 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91475.log 2024-09-14T07:01:46.359 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.76817.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76817.log.gz 2024-09-14T07:01:46.359 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86752.log 2024-09-14T07:01:46.363 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.91475.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91475.log.gz 2024-09-14T07:01:46.364 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85055.log 2024-09-14T07:01:46.366 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86752.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86752.log.gz 2024-09-14T07:01:46.369 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61603.log.gz 2024-09-14T07:01:46.369 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75600.log 2024-09-14T07:01:46.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85055.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85055.log.gz 2024-09-14T07:01:46.370 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87337.log 2024-09-14T07:01:46.370 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84236.log 2024-09-14T07:01:46.372 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87337.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87337.log.gz 2024-09-14T07:01:46.372 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46132.log 2024-09-14T07:01:46.374 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84236.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84236.log.gz 2024-09-14T07:01:46.375 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90242.log 2024-09-14T07:01:46.375 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75600.log.gz 2024-09-14T07:01:46.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.88845.log 2024-09-14T07:01:46.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73998.log 2024-09-14T07:01:46.375 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.0.88845.log: 19.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.88845.log.gz 2024-09-14T07:01:46.375 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48986.log 2024-09-14T07:01:46.377 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.73998.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73998.log.gz 2024-09-14T07:01:46.378 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73096.log 2024-09-14T07:01:46.380 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.48986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55638.log 2024-09-14T07:01:46.380 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90242.log.gz 2024-09-14T07:01:46.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.73096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37037.log 2024-09-14T07:01:46.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.55638.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55638.log.gz 2024-09-14T07:01:46.381 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73096.log.gz 2024-09-14T07:01:46.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38089.log 2024-09-14T07:01:46.381 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.37037.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37037.log.gz 2024-09-14T07:01:46.381 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81749.log 2024-09-14T07:01:46.383 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.38089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45575.log 2024-09-14T07:01:46.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.81749.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81749.log.gz 2024-09-14T07:01:46.384 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71520.log 2024-09-14T07:01:46.384 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42676.log 2024-09-14T07:01:46.384 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45575.log: /var/log/ceph/cluster1-client.mirror.71520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53141.log 2024-09-14T07:01:46.385 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.42676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46350.log 2024-09-14T07:01:46.385 INFO:teuthology.orchestra.run.smithi170.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71520.log.gz 2024-09-14T07:01:46.388 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.53141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45665.log 2024-09-14T07:01:46.389 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48051.log 2024-09-14T07:01:46.391 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45665.log: gzip -5 --verbose 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50939.log.gz 2024-09-14T07:01:46.391 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/cluster2-client.admin.32631.log 2024-09-14T07:01:46.391 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.48051.log: --verbose -- /var/log/ceph/cluster2-client.mirror.69500.log 2024-09-14T07:01:46.391 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.32631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61981.log 2024-09-14T07:01:46.391 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32631.log.gz 2024-09-14T07:01:46.395 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.69500.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69500.log.gz 2024-09-14T07:01:46.395 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51324.log 2024-09-14T07:01:46.395 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64188.log 2024-09-14T07:01:46.395 INFO:teuthology.orchestra.run.smithi170.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49651.log.gz 2024-09-14T07:01:46.395 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.51324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61561.log 2024-09-14T07:01:46.416 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.64188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86572.log 2024-09-14T07:01:46.418 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.61561.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61561.log.gz 2024-09-14T07:01:46.440 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79459.log.gz 2024-09-14T07:01:46.440 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64982.log 2024-09-14T07:01:46.440 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86572.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86572.log.gz 2024-09-14T07:01:46.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43357.log 2024-09-14T07:01:46.441 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.64982.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64982.log.gz 2024-09-14T07:01:46.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88350.log 2024-09-14T07:01:46.445 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43810.log 2024-09-14T07:01:46.445 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88350.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88350.log.gz 2024-09-14T07:01:46.446 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86797.log 2024-09-14T07:01:46.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.43810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36782.log 2024-09-14T07:01:46.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86797.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86797.log.gz 2024-09-14T07:01:46.446 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76078.log 2024-09-14T07:01:46.446 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.36782.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36782.log.gz 2024-09-14T07:01:46.446 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52622.log 2024-09-14T07:01:46.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76078.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76078.log.gz 2024-09-14T07:01:46.448 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87089.log 2024-09-14T07:01:46.448 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.52622.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52622.log.gz 2024-09-14T07:01:46.450 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63780.log 2024-09-14T07:01:46.450 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87089.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87089.log.gz 2024-09-14T07:01:46.450 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89882.log 2024-09-14T07:01:46.452 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.63780.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63780.log.gz 2024-09-14T07:01:46.453 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45908.log 2024-09-14T07:01:46.453 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.89882.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89882.log.gz 2024-09-14T07:01:46.453 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78371.log 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55099.log 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.78371.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78371.log.gz 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86549.log 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.55099.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55099.log.gz 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87517.log 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86549.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86549.log.gz 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88575.log 2024-09-14T07:01:46.457 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87517.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87517.log.gz 2024-09-14T07:01:46.459 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84853.log 2024-09-14T07:01:46.459 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.88575.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88575.log.gz 2024-09-14T07:01:46.460 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44014.log 2024-09-14T07:01:46.461 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84853.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84853.log.gz 2024-09-14T07:01:46.461 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86127.log 2024-09-14T07:01:46.463 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.44014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92332.log 2024-09-14T07:01:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86127.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86127.log.gz 2024-09-14T07:01:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85541.log 2024-09-14T07:01:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92332.log.gz 2024-09-14T07:01:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90282.log 2024-09-14T07:01:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85541.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85541.log.gz 2024-09-14T07:01:46.467 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85317.log 2024-09-14T07:01:46.467 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.90282.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90282.log.gz 2024-09-14T07:01:46.467 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45685.log 2024-09-14T07:01:46.471 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85317.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85317.log.gz 2024-09-14T07:01:46.474 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46325.log 2024-09-14T07:01:46.475 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.45685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75507.log 2024-09-14T07:01:46.475 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85387.log 2024-09-14T07:01:46.475 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.75507.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75507.log.gz 2024-09-14T07:01:46.475 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36655.log 2024-09-14T07:01:46.478 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85387.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85387.log.gz 2024-09-14T07:01:46.479 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83994.log 2024-09-14T07:01:46.479 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6%/var/log/ceph/cluster2-client.admin.36655.log: -- replaced with /var/log/ceph/cluster2-client.mirror.70399.log.gz 2024-09-14T07:01:46.479 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36655.log.gz 2024-09-14T07:01:46.479 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65904.log 2024-09-14T07:01:46.479 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46434.log 2024-09-14T07:01:46.479 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65904.log: /var/log/ceph/cluster2-client.mirror.83994.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65904.log.gz 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83994.log.gz 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65421.log 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.46434.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73549.log.gz 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42152.log 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.65421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72743.log 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.42152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86177.log 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65421.log.gz 2024-09-14T07:01:46.483 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92287.log 2024-09-14T07:01:46.486 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.86177.log: /var/log/ceph/cluster2-client.mirror.72743.log: gzip -5 --verbose -- 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72743.log.gz 2024-09-14T07:01:46.486 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/cluster2-client.mirror.45485.log 2024-09-14T07:01:46.486 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86177.log.gz 2024-09-14T07:01:46.486 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.92287.log: -- /var/log/ceph/cluster2-client.mirror.85288.log 2024-09-14T07:01:46.486 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.92287.log.gz 2024-09-14T07:01:46.486 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.45485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89634.log 2024-09-14T07:01:46.487 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.85288.log: /var/log/ceph/cluster2-client.mirror.89634.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85288.log.gz 2024-09-14T07:01:46.488 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40865.log 2024-09-14T07:01:46.488 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84904.log 2024-09-14T07:01:46.490 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.40865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76021.log 2024-09-14T07:01:46.491 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.84904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87945.log 2024-09-14T07:01:46.491 INFO:teuthology.orchestra.run.smithi170.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84904.log.gz 2024-09-14T07:01:46.491 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.76021.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76021.log.gz 2024-09-14T07:01:46.495 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66561.log 2024-09-14T07:01:46.495 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.87945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86617.log 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87945.log.gz 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.66561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83266.log 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.mirror.86617.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66561.log.gz 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86617.log.gz 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92689.log 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.mirror.83266.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83266.log.gz 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44081.log 2024-09-14T07:01:46.496 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster2-client.admin.92689.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92689.log.gz 2024-09-14T07:01:46.514 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cluster1-client.admin.44081.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.42152.log.gz 2024-09-14T07:01:46.578 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45755.log.gz 2024-09-14T07:01:46.583 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61981.log.gz 2024-09-14T07:01:46.587 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45399.log.gz 2024-09-14T07:01:46.639 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40143.log.gz 2024-09-14T07:01:46.641 INFO:teuthology.orchestra.run.smithi170.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46132.log.gz 2024-09-14T07:01:46.643 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46325.log.gz 2024-09-14T07:01:46.650 INFO:teuthology.orchestra.run.smithi170.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48051.log.gz 2024-09-14T07:01:46.650 INFO:teuthology.orchestra.run.smithi170.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46434.log.gz 2024-09-14T07:01:46.658 INFO:teuthology.orchestra.run.smithi170.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42676.log.gz 2024-09-14T07:01:46.676 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90032.log.gz 2024-09-14T07:01:46.710 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53141.log.gz 2024-09-14T07:01:46.712 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45775.log.gz 2024-09-14T07:01:46.713 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64188.log.gz 2024-09-14T07:01:46.713 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45861.log.gz 2024-09-14T07:01:46.713 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45728.log.gz 2024-09-14T07:01:46.745 INFO:teuthology.orchestra.run.smithi170.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38089.log.gz 2024-09-14T07:01:46.745 INFO:teuthology.orchestra.run.smithi170.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40865.log.gz 2024-09-14T07:01:46.746 INFO:teuthology.orchestra.run.smithi170.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.44081.log.gz 2024-09-14T07:01:46.747 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43357.log.gz 2024-09-14T07:01:46.747 INFO:teuthology.orchestra.run.smithi170.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43810.log.gz 2024-09-14T07:01:46.747 INFO:teuthology.orchestra.run.smithi170.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46350.log.gz 2024-09-14T07:01:46.757 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51324.log.gz 2024-09-14T07:01:46.817 INFO:teuthology.orchestra.run.smithi170.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40979.log.gz 2024-09-14T07:01:46.829 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45685.log.gz 2024-09-14T07:01:46.833 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45376.log.gz 2024-09-14T07:01:46.834 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45485.log.gz 2024-09-14T07:01:46.850 INFO:teuthology.orchestra.run.smithi170.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38577.log.gz 2024-09-14T07:01:46.878 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89634.log.gz 2024-09-14T07:01:46.897 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42360.log.gz 2024-09-14T07:01:46.968 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42407.log.gz 2024-09-14T07:01:47.063 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45575.log.gz 2024-09-14T07:01:47.110 INFO:teuthology.orchestra.run.smithi170.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41340.log.gz 2024-09-14T07:01:47.245 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45908.log.gz 2024-09-14T07:01:47.290 INFO:teuthology.orchestra.run.smithi001.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-14T07:01:47.582 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45665.log.gz 2024-09-14T07:01:49.046 INFO:teuthology.orchestra.run.smithi170.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41636.log.gz 2024-09-14T07:01:51.110 INFO:teuthology.orchestra.run.smithi170.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38639.log.gz 2024-09-14T07:01:53.140 INFO:teuthology.orchestra.run.smithi170.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38577.log.gz 2024-09-14T07:01:53.370 INFO:teuthology.orchestra.run.smithi170.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48074.log.gz 2024-09-14T07:01:55.846 INFO:teuthology.orchestra.run.smithi170.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41819.log.gz 2024-09-14T07:01:56.297 INFO:teuthology.orchestra.run.smithi170.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47970.log.gz 2024-09-14T07:01:57.057 INFO:teuthology.orchestra.run.smithi170.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44109.log.gz 2024-09-14T07:01:58.719 INFO:teuthology.orchestra.run.smithi170.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47689.log.gz 2024-09-14T07:01:58.719 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47823.log.gz 2024-09-14T07:01:59.899 INFO:teuthology.orchestra.run.smithi170.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47363.log.gz 2024-09-14T07:02:01.218 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46952.log.gz 2024-09-14T07:02:01.250 INFO:teuthology.orchestra.run.smithi170.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47599.log.gz 2024-09-14T07:02:01.788 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41452.log.gz 2024-09-14T07:02:02.434 INFO:teuthology.orchestra.run.smithi170.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42452.log.gz 2024-09-14T07:02:03.014 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42917.log.gz 2024-09-14T07:02:03.161 INFO:teuthology.orchestra.run.smithi170.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46890.log.gz 2024-09-14T07:02:05.109 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48733.log.gz 2024-09-14T07:02:06.454 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44489.log.gz 2024-09-14T07:02:06.676 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48538.log.gz 2024-09-14T07:02:06.683 INFO:teuthology.orchestra.run.smithi170.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39009.log.gz 2024-09-14T07:02:06.780 INFO:teuthology.orchestra.run.smithi170.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42180.log.gz 2024-09-14T07:02:07.328 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41002.log.gz 2024-09-14T07:02:07.813 INFO:teuthology.orchestra.run.smithi170.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42537.log.gz 2024-09-14T07:02:07.882 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47053.log.gz 2024-09-14T07:02:09.895 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41087.log.gz 2024-09-14T07:02:09.950 INFO:teuthology.orchestra.run.smithi170.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-14T07:02:10.472 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47445.log.gz 2024-09-14T07:02:11.157 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42091.log.gz 2024-09-14T07:02:11.639 INFO:teuthology.orchestra.run.smithi170.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48190.log.gz 2024-09-14T07:02:11.727 INFO:teuthology.orchestra.run.smithi170.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46733.log.gz 2024-09-14T07:02:11.972 INFO:teuthology.orchestra.run.smithi170.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48986.log.gz 2024-09-14T07:02:14.132 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44382.log.gz 2024-09-14T07:02:14.852 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46667.log.gz 2024-09-14T07:02:15.116 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49273.log.gz 2024-09-14T07:02:16.243 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40641.log.gz 2024-09-14T07:02:16.941 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40730.log.gz 2024-09-14T07:02:17.021 INFO:teuthology.orchestra.run.smithi170.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44014.log.gz 2024-09-14T07:02:17.300 INFO:teuthology.orchestra.run.smithi170.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41363.log.gz 2024-09-14T07:02:20.205 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47260.log.gz 2024-09-14T07:02:20.933 INFO:teuthology.orchestra.run.smithi170.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38916.log.gz 2024-09-14T07:02:22.730 INFO:teuthology.orchestra.run.smithi170.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48360.log.gz 2024-09-14T07:02:24.094 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47142.log.gz 2024-09-14T07:02:27.043 INFO:teuthology.orchestra.run.smithi170.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41724.log.gz 2024-09-14T07:02:27.397 INFO:teuthology.orchestra.run.smithi170.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42813.log.gz 2024-09-14T07:02:41.348 INFO:teuthology.orchestra.run.smithi170.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38516.log.gz 2024-09-14T07:03:33.215 INFO:teuthology.orchestra.run.smithi170.stderr: 94.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38472.log.gz 2024-09-14T07:03:45.785 INFO:teuthology.orchestra.run.smithi170.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38516.log.gz 2024-09-14T07:04:42.589 INFO:teuthology.orchestra.run.smithi001.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-14T07:04:54.420 INFO:teuthology.orchestra.run.smithi001.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-14T07:05:03.227 INFO:teuthology.orchestra.run.smithi170.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38472.log.gz 2024-09-14T07:05:11.746 INFO:teuthology.orchestra.run.smithi001.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-14T07:05:11.748 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-14T07:05:11.748 INFO:teuthology.orchestra.run.smithi001.stderr:real 3m28.226s 2024-09-14T07:05:11.748 INFO:teuthology.orchestra.run.smithi001.stderr:user 3m37.470s 2024-09-14T07:05:11.748 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m8.866s 2024-09-14T07:16:39.282 INFO:teuthology.orchestra.run.smithi170.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-14T07:17:13.326 INFO:teuthology.orchestra.run.smithi170.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-14T07:17:43.791 INFO:teuthology.orchestra.run.smithi170.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-14T07:17:43.793 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-09-14T07:17:43.793 INFO:teuthology.orchestra.run.smithi170.stderr:real 16m0.230s 2024-09-14T07:17:43.794 INFO:teuthology.orchestra.run.smithi170.stderr:user 5m44.992s 2024-09-14T07:17:43.794 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m20.141s 2024-09-14T07:17:43.794 INFO:tasks.ceph:Archiving logs... 2024-09-14T07:17:43.795 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/remote/smithi001/log 2024-09-14T07:17:43.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T07:18:13.432 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/remote/smithi170/log 2024-09-14T07:18:13.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T07:19:01.281 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-14T07:19:01.293 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-14T07:19:01.294 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T07:19:01.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T07:19:01.403 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-14T07:19:01.411 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-14T07:19:01.411 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T07:19:01.414 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T07:19:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-09-14T07:19:01.431 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-09-14T07:19:01.462 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T07:19:01.462 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-09-14T07:19:01.462 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv01.front.sepia.ceph.com 3 6 377 58 -626us[ -622us] +/- 44ms 2024-09-14T07:19:01.463 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 +813us[ +813us] +/- 59ms 2024-09-14T07:19:01.463 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +123us[ +123us] +/- 66ms 2024-09-14T07:19:01.463 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T07:19:01.466 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T07:19:01.467 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-09-14T07:19:01.467 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv01.front.sepia.ceph.com 3 6 377 57 -646us[ -641us] +/- 44ms 2024-09-14T07:19:01.467 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 57 +784us[ +784us] +/- 59ms 2024-09-14T07:19:01.467 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +81us[ +81us] +/- 66ms 2024-09-14T07:19:01.467 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T07:19:01.468 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-14T07:19:01.476 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-14T07:19:01.477 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-14T07:19:01.485 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T07:19:01.587 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T07:19:01.694 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T07:19:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T07:19:01.725 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T07:19:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T07:19:01.756 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-14T07:19:01.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-14T07:19:01.774 INFO:teuthology.task.internal:Duration was 3455.458451 seconds 2024-09-14T07:19:01.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-14T07:19:01.782 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-14T07:19:01.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T07:19:01.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T07:19:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T07:19:01.864 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T07:19:02.323 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-14T07:19:02.324 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-09-14T07:19:02.324 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T07:19:02.348 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-09-14T07:19:02.349 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T07:19:02.376 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-14T07:19:02.376 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T07:19:02.392 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T07:19:02.445 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-14T07:19:02.445 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T07:19:02.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T07:19:02.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-14T07:19:02.618 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-14T07:19:02.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T07:19:02.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T07:19:02.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-14T07:19:02.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T07:19:02.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T07:19:02.727 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-09-14T07:19:02.750 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-09-14T07:19:02.769 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T07:19:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T07:19:02.805 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T07:19:02.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T07:19:02.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-14T07:19:02.834 INFO:teuthology.task.internal:Transferring archived files... 2024-09-14T07:19:02.835 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/remote/smithi001 2024-09-14T07:19:02.835 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T07:19:02.881 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900380/remote/smithi170 2024-09-14T07:19:02.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T07:19:02.914 INFO:teuthology.task.internal:Removing archive directory... 2024-09-14T07:19:02.914 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T07:19:02.917 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T07:19:02.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-14T07:19:02.975 INFO:teuthology.task.internal:Not uploading archives. 2024-09-14T07:19:02.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-14T07:19:02.985 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-14T07:19:02.985 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T07:19:02.987 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T07:19:03.000 INFO:teuthology.orchestra.run.smithi001.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 07:19 /home/ubuntu/cephtest 2024-09-14T07:19:03.018 INFO:teuthology.orchestra.run.smithi170.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 07:19 /home/ubuntu/cephtest 2024-09-14T07:19:03.020 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-14T07:19:03.029 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-14T07:19:03.050 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} duration: 3455.458451271057 failure_reason: '"2024-09-14T06:31:04.474787+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-09-14T07:19:03.050 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T07:19:03.139 INFO:teuthology.run:FAIL